Skip to main content
SpringerLink
Log in

Mutual information and redundancy for categorical data

  • Regular Article
  • Published:
Statistical Papers Aims and scope Submit manuscript

Abstract

Most methods for describing the relationship among random variables require specific probability distributions and some assumptions concerning random variables. Mutual information, based on entropy to measure the dependency among random variables, does not need any specific distribution and assumptions. Redundancy, which is an analogous version of mutual information, is also proposed as a method. In this paper, the concepts of redundancy and mutual information are explored as applied to multi-dimensional categorical data. We found that mutual information and redundancy for categorical data can be expressed as a function of the generalized likelihood ratio statistic under several kinds of independent log-linear models. As a consequence, mutual information and redundancy can also be used to analyze contingency tables stochastically. Whereas the generalized likelihood ratio statistic to test the goodness-of-fit of the log-linear models is sensitive to the sample size, the redundancy for categorical data does not depend on sample size but depends on its cell probabilities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Agresti A (1996) An introduction to categorical data analysis. Wiley, New York

    MATH  Google Scholar 

  • Abramson N (1963) Information theory and coding. Mcgraw Hill, New York

    Google Scholar 

  • Bishop YMM, Fienberg SE, Holland PW (1975) Discrete multivariate analysis. MIT Press, Cambridge

    MATH  Google Scholar 

  • Brillinger R (2004) Some data analyses using mutual information. Braz J Probab Stat 18: 163–183

    MathSciNet  Google Scholar 

  • Brillinger R, Guha A (2007) Mutual information in the frequency domain. J Stat Plan Inference 137(3): 1076–1084

    Article  MATH  MathSciNet  Google Scholar 

  • Cover T, Thomas J (1991) Elements of information theory. Wiley, New York

    Book  MATH  Google Scholar 

  • Christensen R (1997) Log-linear models and logistic regression, 2nd edn. Springer, Heidelberg

    MATH  Google Scholar 

  • Darbellay GA (1999) An estimator for the mutual information based on a criterion for conditional independence. Comput Stat Data Anal 32(1): 1–17

    Article  MathSciNet  Google Scholar 

  • DeGroot MH (1962) Uncertainty, information and sequential experiments. Ann Math Stat 33: 404–419

    Article  MATH  MathSciNet  Google Scholar 

  • Fraser A, Swinney H (1986) Independent coordinates for strange attractors from mutual information. Phys Rev 33(2): 1134–1140

    MATH  MathSciNet  Google Scholar 

  • Gallager RG (1968) Information theory and reliable communication. Wiley, New York

    MATH  Google Scholar 

  • Joe H (1989a) Relative entropy measures of multivariate dependence. J Am Stat Assoc 84: 157–164

    Article  MATH  MathSciNet  Google Scholar 

  • Joe H (1989b) Estimation for entropy and other functionals of a multivariate density. Ann Inst Stat Math 41: 683–697

    Article  MATH  MathSciNet  Google Scholar 

  • Moddemeijer R (1989) On estimation of entropy and mutual information of continuous distributions. Signal Process 16(3): 233–246

    Article  MathSciNet  Google Scholar 

  • Palus M (1993) Identifying and quantifying chaos by using information theoretic functions in time series prediction: forecasting the future and understanding the past. SantaFe Inst Stud Sci Complexity 15: 387–413

    Google Scholar 

  • Palus M, Pivka D (1995) Estimating predictability: redundancy and surrogate data method. Neural Network World 15: 537–552

    Google Scholar 

  • Palus M, Vejmelka M (2007) Directionality of coupling from bivariate time series: how to avoid false causalities and missed connections. Phys Rev E 17: 056211

    Article  MathSciNet  Google Scholar 

  • Prichard D, Theiler J (1994) Generating surrogate data for time series with several simultaneously measured variables. Phys Rev Lett 73: 951–954

    Article  Google Scholar 

  • Roulston M (1997) Significance testing of information theoretic functionals. Phys D Nonlinear Phenomena 110: 62–66

    Article  MATH  Google Scholar 

  • Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27: 379–423

    MATH  MathSciNet  Google Scholar 

  • Wienholt W, Sendhoff B (1996) How to determine the redundancy of noisy chaotic time series. Int J Bifurcation Chaos 6: 101–117

    Article  MATH  MathSciNet  Google Scholar 

  • Yule GU (1900) On the association of attributes in statistics. Philos Trans Ser A 194: 257–319

    Article  Google Scholar 

  • Yule GU (1912) On the methods of measuring association between two attributes. J R Stat Soc 75: 579–642

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Statistics, Sungkyunkwan University, Seoul, 110-745, Korea

    Chong Sun Hong

  2. Risk Management, Korea Ratings, Seoul, 150-737, Korea

    Beom Jun Kim

Authors
  1. Chong Sun Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beom Jun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chong Sun Hong.

Additional information

This paper was supported by Samsung Research Fund, Sunkyunkwan University, 2006.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hong, C.S., Kim, B.J. Mutual information and redundancy for categorical data. Stat Papers 52, 17–31 (2011). https://doi.org/10.1007/s00362-009-0196-x

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00362-009-0196-x

Keywords

Search

Navigation