Measuring Distances Between Variables by Mutual Information
Information theoretic concepts, such as the mutual information, provide a general framework to detect and evaluate dependencies between variables. In this work, we describe and review several aspects of the mutual information as a measure of ‘distance’ between variables. Giving a brief overview over the mathematical background, including its recent generalization in the sense of Tsallis, our emphasis will be the numerical estimation of these quantities from finite datasets. The described concepts will be exemplified using large-scale gene expression data and compared to the results obtained from other measures, such as the Pearson Correlation.
KeywordsMutual Information Shannon Entropy Kernel Density Estimator Mutual Infor Finite Data
Unable to display preview. Download preview PDF.
- BUTTE, A.J. and KOHANE, I.S. (2000): Mutual information relevance networks: Functional genomic clustering using pairwise entropy measurements. Pacific Symposium on Biocomputing, 5, 415–426.Google Scholar
- COVER, T.M. and THOMAS, J.A. (1991): Elements of Information Theory. John Wiley, New York.Google Scholar
- LIANG, S, FUHRMAN, S., and SOMOGYI, R. (1998): Reveal, a general reverse engineering algorithm for inference of genetic network architectures. Pacific Symposium on Biocomputing, 3, 18–29.Google Scholar
- MICHAELS, G.S., CARR, D.B., ASKENAZI, M., FUHRMAN, S., WEN, X., and SOMOGYI, R. (1998): Cluster analysis and data visualization of large-scale gene expression data. Pacific Symposium on Biocomputing, 3, 42–53.Google Scholar
- PRESS, W.H., TEUKOLSKY, S.A., VETTERLING, W.T., and FLANNERY, B.P. (1992): Numerical Recipes in C. Second edition, Cambridge University Press, Cambridge.Google Scholar
- SCHENA, M., SHALON, D., DAVIS, R.W., and BROWN, P.O. (1995): Quantitative monitoring of gene expression patterns with a complementary DNA microarray. Science, 270, 467–470.Google Scholar
- SILVERMAN, B.W. (1986): Density Estimation for Statistics and Data Analysis. Chapmann and Hall, London.Google Scholar
- SOMOGYI, R., FUHRMAN, S., and WEN, X. (2001): Genetic network inference in computational models and applications to large-scale gene expression data. In: J. M. Bower and H. Bolouri (Eds.): Computational Modeling of Genetic and Biochemical Networks. MIT Press, Cambridge, 129–157.Google Scholar
- STEUER, R., KURTHS, J, DAUB, CO, WEISE, J, and SELBIG, J. (2002): The mutual information: Detecting and evaluating dependencies between variables. Bioinformatics, 18(Suppl. 2), 231–240.Google Scholar