Measuring Distances Between Variables by Mutual Information

  • Ralf Steuer
  • Carsten O. Daub
  • Joachim Selbig
  • Jürgen Kurths
Part of the Studies in Classification, Data Analysis, and Knowledge Organization book series (STUDIES CLASS)


Information theoretic concepts, such as the mutual information, provide a general framework to detect and evaluate dependencies between variables. In this work, we describe and review several aspects of the mutual information as a measure of ‘distance’ between variables. Giving a brief overview over the mathematical background, including its recent generalization in the sense of Tsallis, our emphasis will be the numerical estimation of these quantities from finite datasets. The described concepts will be exemplified using large-scale gene expression data and compared to the results obtained from other measures, such as the Pearson Correlation.


Mutual Information Shannon Entropy Kernel Density Estimator Mutual Infor Finite Data 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. BRAZMA, A. and VILO J. (2000): Gene expression data analysis. FEBS Letters, 480, 17–24.CrossRefGoogle Scholar
  2. BUTTE, A.J. and KOHANE, I.S. (2000): Mutual information relevance networks: Functional genomic clustering using pairwise entropy measurements. Pacific Symposium on Biocomputing, 5, 415–426.Google Scholar
  3. COVER, T.M. and THOMAS, J.A. (1991): Elements of Information Theory. John Wiley, New York.Google Scholar
  4. CURADO, E.M.F. and TSALLIS, C. (1991): Generalized statistical mechanics: Connection with thermodynamics. J. Phys. A, 24, L69.MathSciNetCrossRefGoogle Scholar
  5. D’HAESELEER, P., LIANG, S., and SOMOGYI, R. (2000): Genetic network inference: From co-expression clustering to reverse engineering. Bioinformatics, 16(8), 707–726.CrossRefGoogle Scholar
  6. EISEN, M.B, SPELLMAN, P.T., BROWN, P.O., and BOTSTEIN, D. (1998): Cluster analysis and display of genome-wide expression patterns. Proc. Natl. Acad. Sci. USA, 95, 14863–14868.CrossRefGoogle Scholar
  7. FRASER, A.M. and SWINNEY, H.L. (1986): Independent coordinates for strange attractors from mutual information. Phys. Rev. A, 33(2), 2318–2321.MathSciNetCrossRefGoogle Scholar
  8. GROSSE, I., HERZEL, H., BULDYREV, S.V., and STANLEY, H.E. (2000): Species independence of mutual information in coding and noncoding DNA. Phys. Rev. E, 61(5), 5624–5629.CrossRefGoogle Scholar
  9. HERWIG, R., POUSTKA, A.J., MUELLER, C, BULL, C, LEHRACH, H., and O’BRIAN, J. (1999): Large-scale clustering of cDNA-fingerprinting data. Genome Research, 9(11), 1093–1105.CrossRefGoogle Scholar
  10. HERZEL, H. and GROSSE, I. (1995): Measuring correlations in symbols sequences. Physica A, 216, 518–542.MathSciNetCrossRefGoogle Scholar
  11. HERZEL, H. and GROSSE, I. (1997): Correlations in DNA sequences: The role of protein coding segments. Phys. Rev. E, 55(1), 800–810.CrossRefGoogle Scholar
  12. HERZEL, H., SCHMITT, A.O., and EBELING, W. (1994): Finite sample effects in sequence analysis. Chaos, Solitons & Fractals, 4(1), 97–113.CrossRefGoogle Scholar
  13. HUGHES, T.R. et al. (2000): Functional discovery via a compendium of expression profiles. Cell, 102, 109–126.CrossRefGoogle Scholar
  14. LIANG, S, FUHRMAN, S., and SOMOGYI, R. (1998): Reveal, a general reverse engineering algorithm for inference of genetic network architectures. Pacific Symposium on Biocomputing, 3, 18–29.Google Scholar
  15. MICHAELS, G.S., CARR, D.B., ASKENAZI, M., FUHRMAN, S., WEN, X., and SOMOGYI, R. (1998): Cluster analysis and data visualization of large-scale gene expression data. Pacific Symposium on Biocomputing, 3, 42–53.Google Scholar
  16. MOON, Y., RAJAGOPALAN, B., and LALL, U. (1995): Estimation of mutual information using kernel density estimators. Phys. Rev. E, 52(3), 2318–2321.CrossRefGoogle Scholar
  17. PRESS, W.H., TEUKOLSKY, S.A., VETTERLING, W.T., and FLANNERY, B.P. (1992): Numerical Recipes in C. Second edition, Cambridge University Press, Cambridge.Google Scholar
  18. SCHENA, M., SHALON, D., DAVIS, R.W., and BROWN, P.O. (1995): Quantitative monitoring of gene expression patterns with a complementary DNA microarray. Science, 270, 467–470.Google Scholar
  19. SHANNON, C.E. (1948): A mathematical theory of communication. The Bell System Technical Journal, 27, 379–423, ibid. 623–656.MathSciNetGoogle Scholar
  20. SILVERMAN, B.W. (1986): Density Estimation for Statistics and Data Analysis. Chapmann and Hall, London.Google Scholar
  21. SOMOGYI, R., FUHRMAN, S., and WEN, X. (2001): Genetic network inference in computational models and applications to large-scale gene expression data. In: J. M. Bower and H. Bolouri (Eds.): Computational Modeling of Genetic and Biochemical Networks. MIT Press, Cambridge, 129–157.Google Scholar
  22. STEUER, R., KURTHS, J, DAUB, CO, WEISE, J, and SELBIG, J. (2002): The mutual information: Detecting and evaluating dependencies between variables. Bioinformatics, 18(Suppl. 2), 231–240.Google Scholar
  23. TSALLIS, C. (1998): Generalized entropy-based criterion for consistent testing. Phys. Rev. E, 58(2), 1442–1445.MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin · Heidelberg 2005

Authors and Affiliations

  • Ralf Steuer
    • 1
  • Carsten O. Daub
    • 2
  • Joachim Selbig
    • 2
  • Jürgen Kurths
    • 1
  1. 1.Nonlinear Dynamics GroupUniversity of PotsdamPotsdamGermany
  2. 2.Max-Planck Institute for Molecular Plant PhysiologyGolmGermany

Personalised recommendations