Statistical Estimation of the Shannon Entropy
- 86 Downloads
The behavior of the Kozachenko–Leonenko estimates for the (differential) Shannon entropy is studied when the number of i.i.d. vector-valued observations tends to infinity. The asymptotic unbiasedness and L2-consistency of the estimates are established. The conditions employed involve the analogues of the Hardy–Littlewood maximal function. It is shown that the results are valid in particular for the entropy estimation of any nondegenerate Gaussian vector.
KeywordsShannon differential entropy Kozachenko–Leonenko estimates Hardy–Littlewood maximal function analogues asymptotic unbiasedness and L2-consistency Gaussian vectors
MR(2010) Subject Classification60F25 62G20 62H12
Unable to display preview. Download preview PDF.
The work is supported by the Russian Science Foundation under grant 14-21-00162 and performed at the Steklov Mathematical Institute of Russian Academy of Sciences. The authors are grateful to Professor E. Spodarev for drawing their attention to the entropy estimation problems.
- Alonso–Ruiz, P., Spodarev, E.: Entropy–based inhomogeneity detection in porous media, arXiv:1611.02241Google Scholar
- Delattre, S., Fournier, N.: On the Kozachenko–Leonenko entropy estimator. Journal of Statistical Planning and Inference, (2017), DOI: http://dx.doi.org/10.1016/j.jspi.2017.01.004 (accepted manuscript)Google Scholar
- Miller, E. G.: A new class of entropy estimators for multidimensional densities. In: Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP03), Hong Kong, China, 6–10 April 2003, 3, 297–300Google Scholar
- Pál, D., Póczos, B., Szepesvári, C.: Estimation of Rényi entropy and mutual information based on generalized nearest–neighbor graphs. In: NIPS’10 Proceedings of the 23rd International Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada (December 06–09, 2010), 1849–1857Google Scholar
- Shannon, C. E.: A mathematical theory of communication. Bell Systems Technical Journal, 27, July and October, 379–423 and 623–656 (1948)Google Scholar
- Singh, S., Pószoc, B.: Nonparanormal information estimation. In: Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 3210–3219, 2017Google Scholar
- Singh, S., Pószoc, B.: Analysis of k–nearest neighbor distances with application to entropy estimation, arXiv: 1603.08578v2Google Scholar