Kolmogorov Complexity, Data Compression, and Inference

Abstract

If a sequence of random variables has Shannon entropy H, it is well known that there exists an efficient description of this sequence which requires only H bits. But the entropy H of a sequence also has to do with inference. Low entropy sequences allow good guesses of their next terms. This is best illustrated by allowing a gambler to gamble at fair odds on such a sequence. The amount of money that one can make is essentially the complement of the entropy with respect to the length of the sequence.

Now suppose that the sequence is not random. Although the entropy of such a sequence is not defined, there is a notion of its intrinsic descriptive complexity. This idea, put forth by Kolmogorov, Chaitin, and Solomonoff, says that the intrinsic complexity of a sequence is the length of its shortest description. Here too there is a tradeoff between complexity and inference. Low complexity sequences allow a high degree of inference. Again there is a gambling tradeoff.

Finally, it will be shown that if a sequence is random and has entropy H, then with high probability its Kolmogorov complexity will also be H.

Special attention will be given to the so-called Kolmogorov H function, a function that has not yet made its appearance in the literature. We argue that it plays the role of a minimal sufficient statistic. Thus, we can assert that there is a sufficient statistic for the Mona Lisa. This idea will capture the fundamental structure of geometrical patterns, probability distributions and the laws of nature.