Error estimation for the determination of entropy and information rate from relative frequencies

Summary

Experimental determination of the entropy H and information rate T of experiments or information sources rely on measurements of the relative frequencies of events and thus furnish only approximations to the exact values of H and T, defined by probabilities rather than relative frequencies. We derive an error estimate for the measured entropy and information rate, which depends only upon the number of possible events and not upon the numerical values of their probabilities, and thereby answer the question of how often experiments should be repeated independently in order that the measured entropy and information rate come sufficiently close to the exact values of H and T with probability sufficiently close to 1.