Abstract
We consider the problem of improving the efficiency of the nonparametric entropy estimation for a stationary ergodic process. Our approach is based on the nearest-neighbor distances. We propose a broad class of metrics on the space of right-sided infinite sequences drawn from a finite alphabet. The new metric has a parameter that is a nonincreasing function. We prove that, under certain conditions, our estimators have a small variance and show that a special selection of the metric parameters reduces the estimator’s bias.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
D. Aldous and P. Shields, “A diffusion limit for a class of randomly-growing binary trees,” Probab. Theory Relat. Fields, 79, 509–542 (1988).
L. Devroye, “Exponential inequalities in nonparametric estimation,” in G. Roussas, ed., Nonparametric Functional Estimation and Related Topics, NATO ASI Ser., Kluwer Academic, Dordrecht (1991), pp. 31–44.
M. Deza and T. Deza, Encyclopedia of Distances, Springer, Berlin (2009).
I. S. Gradshtein and I. M. Ryzhik, Table of Integrals, Series, and Products, Academic Press, London (1994).
P. Grassberger, “Estimating the information content of symbol sequences and efficient codes,” IEEE Trans. Inform. Theory, 35, 669–675 (1989).
A. Kaltchenko and N. Timofeeva, “Entropy estimators with almost sure convergence and an O(n −1) variance,” Adv. Math. Commun., 2, 1–13 (2008).
A. Kaltchenko and N. Timofeeva, “Rate of convergence of the nearest neighbor entropy estimator,” AEU — Int. J. Electron. Commun., 64, 75–79 (2010).
D. E. Knuth, The Art of Computer Programming, Vol. 3, Sorting and Searching, Addison-Wesley, Reading, MA (1975).
I. Kontoyiannis and Yu. M. Suhov, “Prefixes and the entropy rate for long-range sources,” in: F. P. Kelly, ed., Probability Statistics and Optimization, Wiley, New York (1994), pp. 89–98.
N. Martin and J. England, Mathematical Theory of Entropy, Cambridge Univ. Press, Cambridge (1984).
C. McDiarmid, “On the method of bounded differences,” in: Surveys in Combinatorics, Cambridge Univ. Press, Cambridge (1989), pp. 148–188.
P. C. Shields, “Entropy and prefixes,” Ann. Probab., 20, 403–409 (1992).
M. J. Silvapulle and P. K. Sen, Constrained Statistical Inference: Inequality, Order and Shape Restrictions, Wiley, New York (2005).
E. A. Timofeev, “Statistical estimation of measure invariants,” St. Petersburg Math. J., 17, No. 3, 527–551 (2006).
E. A. Timofeev, “Bias of a nonparametric entropy estimator for Markov measures,” J. Math. Sci., 176, No. 2, 255–269 (2011).
J. Ziv and A. Lempel, “Compression of individual sequences by variable rate coding,” IEEE Trans. Inform. Theory, 24, 530–536 (1978).
Author information
Authors and Affiliations
Corresponding author
Additional information
Translated from Fundamentalnaya i Prikladnaya Matematika, Vol. 18, No. 2, pp. 209–227, 2013.
Rights and permissions
About this article
Cite this article
Timofeev, E. Selection of a Metric for the Nearest Neighbor Entropy Estimators. J Math Sci 203, 892–906 (2014). https://doi.org/10.1007/s10958-014-2181-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10958-014-2181-8