Asymptotic Log-Loss of Prequential Maximum Likelihood Codes

Abstract

We analyze the Dawid-Rissanen prequential maximum likelihood codes relative to one-parameter exponential family models ${\mathcal M}$. If data are i.i.d. according to an (essentially) arbitraryP, then the redundancy grows at rate ${\frac{1}{2}} {\rm c} {\rm ln} n$. We show that c = σ $_{\rm 1}^{\rm 2}$/ σ $_{\rm 2}^{\rm 2}$, where σ $_{\rm 1}^{\rm 2}$ is the variance of P, and σ $_{\rm 2}^{\rm 2}$ is the variance of the distribution $M^{*} \in {\mathcal M}$ that is closest to P in KL divergence. This shows that prequential codes behave quite differently from other important universal codes such as the 2-part MDL, Shtarkov and Bayes codes, for which c = 1. This behavior is undesirable in an MDL model selection setting.