Skip to main content

Selection of a Metric for the Nearest Neighbor Entropy Estimators


We consider the problem of improving the efficiency of the nonparametric entropy estimation for a stationary ergodic process. Our approach is based on the nearest-neighbor distances. We propose a broad class of metrics on the space of right-sided infinite sequences drawn from a finite alphabet. The new metric has a parameter that is a nonincreasing function. We prove that, under certain conditions, our estimators have a small variance and show that a special selection of the metric parameters reduces the estimator’s bias.

This is a preview of subscription content, access via your institution.


  1. 1.

    D. Aldous and P. Shields, “A diffusion limit for a class of randomly-growing binary trees,” Probab. Theory Relat. Fields, 79, 509–542 (1988).

    Article  MATH  MathSciNet  Google Scholar 

  2. 2.

    L. Devroye, “Exponential inequalities in nonparametric estimation,” in G. Roussas, ed., Nonparametric Functional Estimation and Related Topics, NATO ASI Ser., Kluwer Academic, Dordrecht (1991), pp. 31–44.

  3. 3.

    M. Deza and T. Deza, Encyclopedia of Distances, Springer, Berlin (2009).

    Book  MATH  Google Scholar 

  4. 4.

    I. S. Gradshtein and I. M. Ryzhik, Table of Integrals, Series, and Products, Academic Press, London (1994).

    Google Scholar 

  5. 5.

    P. Grassberger, “Estimating the information content of symbol sequences and efficient codes,” IEEE Trans. Inform. Theory, 35, 669–675 (1989).

    Article  MathSciNet  Google Scholar 

  6. 6.

    A. Kaltchenko and N. Timofeeva, “Entropy estimators with almost sure convergence and an O(n 1) variance,” Adv. Math. Commun., 2, 1–13 (2008).

    Article  MATH  MathSciNet  Google Scholar 

  7. 7.

    A. Kaltchenko and N. Timofeeva, “Rate of convergence of the nearest neighbor entropy estimator,” AEU — Int. J. Electron. Commun., 64, 75–79 (2010).

    Article  Google Scholar 

  8. 8.

    D. E. Knuth, The Art of Computer Programming, Vol. 3, Sorting and Searching, Addison-Wesley, Reading, MA (1975).

  9. 9.

    I. Kontoyiannis and Yu. M. Suhov, “Prefixes and the entropy rate for long-range sources,” in: F. P. Kelly, ed., Probability Statistics and Optimization, Wiley, New York (1994), pp. 89–98.

    Google Scholar 

  10. 10.

    N. Martin and J. England, Mathematical Theory of Entropy, Cambridge Univ. Press, Cambridge (1984).

    Book  Google Scholar 

  11. 11.

    C. McDiarmid, “On the method of bounded differences,” in: Surveys in Combinatorics, Cambridge Univ. Press, Cambridge (1989), pp. 148–188.

  12. 12.

    P. C. Shields, “Entropy and prefixes,” Ann. Probab., 20, 403–409 (1992).

    Article  MATH  MathSciNet  Google Scholar 

  13. 13.

    M. J. Silvapulle and P. K. Sen, Constrained Statistical Inference: Inequality, Order and Shape Restrictions, Wiley, New York (2005).

    Google Scholar 

  14. 14.

    E. A. Timofeev, “Statistical estimation of measure invariants,” St. Petersburg Math. J., 17, No. 3, 527–551 (2006).

    Article  MATH  MathSciNet  Google Scholar 

  15. 15.

    E. A. Timofeev, “Bias of a nonparametric entropy estimator for Markov measures,” J. Math. Sci., 176, No. 2, 255–269 (2011).

    Article  MATH  MathSciNet  Google Scholar 

  16. 16.

    J. Ziv and A. Lempel, “Compression of individual sequences by variable rate coding,” IEEE Trans. Inform. Theory, 24, 530–536 (1978).

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to E. Timofeev.

Additional information

Translated from Fundamentalnaya i Prikladnaya Matematika, Vol. 18, No. 2, pp. 209–227, 2013.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Timofeev, E. Selection of a Metric for the Nearest Neighbor Entropy Estimators. J Math Sci 203, 892–906 (2014).

Download citation


  • Entropy
  • Auxiliary Parameter
  • Entropy Rate
  • Entropy Estimator
  • Markov Measure