Advertisement

Problems of Information Transmission

, Volume 43, Issue 3, pp 167–189 | Cite as

Normalized information-based divergences

  • J. -F. Coeurjolly
  • R. Drouilhet
  • J. -F. Robineau
Information Theory

Abstract

This paper is devoted to the mathematical study of some divergences based on mutual information which are well suited to categorical random vectors. These divergences are generalizations of the “entropy distance” and “information distance.” Their main characteristic is that they combine a complexity term and the mutual information. We then introduce the notion of (normalized) information-based divergence, propose several examples, and discuss their mathematical properties, in particular, in some prediction framework.

Keywords

Entropy Mutual Information Random Vector Information Transmission Triangle Inequality 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Shannon, C.E., A Mathematical Theory of Communication (continued), Bell Syst. Tech. J., 1948, vol. 27, no. 4, pp. 623–656.MathSciNetGoogle Scholar
  2. 2.
    Crutchfield J.P., Information and Its Metric, Nonlinear Structures in Physical Systems: Pattern Formation, Chaos, and Waves (Proc. 2nd Woodward Conf., San Jose State Univ., 1989), Lam, L. and Morris, H.C., Eds., New York: Springer, 1990, pp. 119–130.Google Scholar
  3. 3.
    Hillman, C., A Formal Theory of Information: I. Statics, Preprint, 1998. Available from http://citeseer.ist.psu.edu/89520.html.
  4. 4.
    Bennett, C.H., Gács, P., Li, M., Vitányi, P.M.B., and Zurek, W.H., Information Distance, IEEE Trans. Inform. Theory, 1998, vol. 44, no. 4, pp. 1407–1423.MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Li, M., Chen, X., Li, X., Ma, B., and Vitányi, P.M.B., The Similarity Metric, IEEE Trans. Inform. Theory, 2004, vol. 50, no. 12, pp. 3250–3264.CrossRefMathSciNetGoogle Scholar
  6. 6.
    Cilibrasi, R. and Vitányi, P.M.B., Clustering by Compression, IEEE Trans. Inform. Theory, 2005, vol. 51, no. 4, pp. 1523–1545.CrossRefMathSciNetGoogle Scholar
  7. 7.
    Li, M., Badger, J.H., Chen, X., Kwong, S., Kearney, P., and Zhang, H., An Information-Based Sequence Distance and Its Application to Whole Mitochondrial Genome Phylogeny, Bioinformatics, 2001, vol. 17, no. 1, pp. 149–154.CrossRefGoogle Scholar
  8. 8.
    Cilibrasi, R., Vitányi, P., and de Wolf, R., Algorithmic Clustering of Music, 2003, e-print arxiv.org/cs.SD/0303025.Google Scholar
  9. 9.
    Cilibrasi, R. and Vitányi, P., Automatic Meaning Discovery Using Google, 2005, e-print arxiv.org/cs.CL/0412098.Google Scholar
  10. 10.
    Grünwald, P. and Vitányi, P., Shannon Information and Kolmogorov Complexity, 2004, e-print arxiv.org/cs.IT/0410002.Google Scholar
  11. 11.
    Leung-Yan-Cheong, S.K. and Cover, T.M., Some Equivalences between Shannon Entropy and Kolmogorov Complexity, IEEE Trans. on Inf. Theory, 1978, vol. 24, no. 3, pp. 331–338.MATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Hammer, D., Romashchenko, A., Shen, A., and Vereshchagin, N., Inequalities for Shannon Entropy and Kolmogorov Complexity, J. Comput. Syst. Sci., 2000, vol. 60, no. 2, pp. 442–464.MATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Kraskov, A., Stögbauer, H., Andrzejak, R.G., and Grassberger, P., Hierarchical Clustering Based on Mutual Information, 2003, e-print arxiv.org/q-bio/0311039.Google Scholar
  14. 14.
    Liu, H. and Motoda, H., Feature Selection for Knowledge Discovery and Data Mining, Boston: Kluwer, 1998.MATHGoogle Scholar
  15. 15.
    Robineau, J.-F., Méthodes de sélection de variables (parmi un grand nombre) dans un cadre de discrimination, PhD Thesis, Grenoble, France: Univ. Joseph Fourier, 2004.Google Scholar
  16. 16.
    Granger, C.W., Maasoumi, E., and Racine, J., A Dependence Metric for Possibly Nonlinear Processes, J. Time Ser. Anal., 2004, vol. 25, no. 5, pp. 649–669.MATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Cover, T.M. and Thomas, J.A., Elements of Information Theory, New York: Wiley, 1991.MATHGoogle Scholar
  18. 18.
    Kaltchenko, A., Algorithms for Estimation of Information Distance with Application to Bioinformatics and Linguistics, in Proc. 2004 Canadian Conf. on Electrical and Computer Engineering, 2004, vol. 4, p. 2255. Available from arxiv.org/cs.CC/0404039.Google Scholar
  19. 19.
    Li, M. and Vitányi, P., An Introduction to Kolmogorov Complexity and Its Applications, New York: Springer, 1997, 2nd ed.MATHGoogle Scholar
  20. 20.
    Li, M, Chen, X., Li, X., Ma, B., and Vitányi, P., The Similarity Metric, in Proc. 14th Annual ACMSIAM Symp. on Discrete Algorithms (SODA-03), New York, 2003, pp. 863–872.Google Scholar
  21. 21.
    Ullah, A., Entropy, Divergence and Distance Measures with Econometric Applications, J. Stat. Plan. Inference, 1996, vol. 49, no. 1, pp. 137–162.MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Pleiades Publishing, Inc. 2007

Authors and Affiliations

  • J. -F. Coeurjolly
    • 1
  • R. Drouilhet
    • 1
  • J. -F. Robineau
    • 1
  1. 1.Université Pierre Mendès-FranceGrenobleFrance

Personalised recommendations