Problems of Information Transmission

, Volume 43, Issue 3, pp 167–189 | Cite as

Normalized information-based divergences

  • J. -F. Coeurjolly
  • R. Drouilhet
  • J. -F. Robineau
Information Theory


This paper is devoted to the mathematical study of some divergences based on mutual information which are well suited to categorical random vectors. These divergences are generalizations of the “entropy distance” and “information distance.” Their main characteristic is that they combine a complexity term and the mutual information. We then introduce the notion of (normalized) information-based divergence, propose several examples, and discuss their mathematical properties, in particular, in some prediction framework.


Entropy Mutual Information Random Vector Information Transmission Triangle Inequality 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Shannon, C.E., A Mathematical Theory of Communication (continued), Bell Syst. Tech. J., 1948, vol. 27, no. 4, pp. 623–656.MathSciNetGoogle Scholar
  2. 2.
    Crutchfield J.P., Information and Its Metric, Nonlinear Structures in Physical Systems: Pattern Formation, Chaos, and Waves (Proc. 2nd Woodward Conf., San Jose State Univ., 1989), Lam, L. and Morris, H.C., Eds., New York: Springer, 1990, pp. 119–130.Google Scholar
  3. 3.
    Hillman, C., A Formal Theory of Information: I. Statics, Preprint, 1998. Available from
  4. 4.
    Bennett, C.H., Gács, P., Li, M., Vitányi, P.M.B., and Zurek, W.H., Information Distance, IEEE Trans. Inform. Theory, 1998, vol. 44, no. 4, pp. 1407–1423.zbMATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Li, M., Chen, X., Li, X., Ma, B., and Vitányi, P.M.B., The Similarity Metric, IEEE Trans. Inform. Theory, 2004, vol. 50, no. 12, pp. 3250–3264.CrossRefMathSciNetGoogle Scholar
  6. 6.
    Cilibrasi, R. and Vitányi, P.M.B., Clustering by Compression, IEEE Trans. Inform. Theory, 2005, vol. 51, no. 4, pp. 1523–1545.CrossRefMathSciNetGoogle Scholar
  7. 7.
    Li, M., Badger, J.H., Chen, X., Kwong, S., Kearney, P., and Zhang, H., An Information-Based Sequence Distance and Its Application to Whole Mitochondrial Genome Phylogeny, Bioinformatics, 2001, vol. 17, no. 1, pp. 149–154.CrossRefGoogle Scholar
  8. 8.
    Cilibrasi, R., Vitányi, P., and de Wolf, R., Algorithmic Clustering of Music, 2003, e-print Scholar
  9. 9.
    Cilibrasi, R. and Vitányi, P., Automatic Meaning Discovery Using Google, 2005, e-print Scholar
  10. 10.
    Grünwald, P. and Vitányi, P., Shannon Information and Kolmogorov Complexity, 2004, e-print Scholar
  11. 11.
    Leung-Yan-Cheong, S.K. and Cover, T.M., Some Equivalences between Shannon Entropy and Kolmogorov Complexity, IEEE Trans. on Inf. Theory, 1978, vol. 24, no. 3, pp. 331–338.zbMATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Hammer, D., Romashchenko, A., Shen, A., and Vereshchagin, N., Inequalities for Shannon Entropy and Kolmogorov Complexity, J. Comput. Syst. Sci., 2000, vol. 60, no. 2, pp. 442–464.zbMATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Kraskov, A., Stögbauer, H., Andrzejak, R.G., and Grassberger, P., Hierarchical Clustering Based on Mutual Information, 2003, e-print Scholar
  14. 14.
    Liu, H. and Motoda, H., Feature Selection for Knowledge Discovery and Data Mining, Boston: Kluwer, 1998.zbMATHGoogle Scholar
  15. 15.
    Robineau, J.-F., Méthodes de sélection de variables (parmi un grand nombre) dans un cadre de discrimination, PhD Thesis, Grenoble, France: Univ. Joseph Fourier, 2004.Google Scholar
  16. 16.
    Granger, C.W., Maasoumi, E., and Racine, J., A Dependence Metric for Possibly Nonlinear Processes, J. Time Ser. Anal., 2004, vol. 25, no. 5, pp. 649–669.zbMATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Cover, T.M. and Thomas, J.A., Elements of Information Theory, New York: Wiley, 1991.zbMATHGoogle Scholar
  18. 18.
    Kaltchenko, A., Algorithms for Estimation of Information Distance with Application to Bioinformatics and Linguistics, in Proc. 2004 Canadian Conf. on Electrical and Computer Engineering, 2004, vol. 4, p. 2255. Available from Scholar
  19. 19.
    Li, M. and Vitányi, P., An Introduction to Kolmogorov Complexity and Its Applications, New York: Springer, 1997, 2nd ed.zbMATHGoogle Scholar
  20. 20.
    Li, M, Chen, X., Li, X., Ma, B., and Vitányi, P., The Similarity Metric, in Proc. 14th Annual ACMSIAM Symp. on Discrete Algorithms (SODA-03), New York, 2003, pp. 863–872.Google Scholar
  21. 21.
    Ullah, A., Entropy, Divergence and Distance Measures with Econometric Applications, J. Stat. Plan. Inference, 1996, vol. 49, no. 1, pp. 137–162.zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Pleiades Publishing, Inc. 2007

Authors and Affiliations

  • J. -F. Coeurjolly
    • 1
  • R. Drouilhet
    • 1
  • J. -F. Robineau
    • 1
  1. 1.Université Pierre Mendès-FranceGrenobleFrance

Personalised recommendations