MDL convergence speed for Bernoulli sequences
 Jan Poland,
 Marcus Hutter
 … show all 2 hide
Rent the article at a discount
Rent now* Final gross prices may vary according to local VAT.
Get AccessAbstract
The Minimum Description Length principle for online sequence estimation/prediction in a proper learning setup is studied. If the underlying model class is discrete, then the total expected square loss is a particularly interesting performance measure: (a) this quantity is finitely bounded, implying convergence with probability one, and (b) it additionally specifies the convergence speed. For MDL, in general one can only have loss bounds which are finite but exponentially larger than those for Bayes mixtures. We show that this is even the case if the model class contains only Bernoulli distributions. We derive a new upper bound on the prediction error for countable Bernoulli classes. This implies a small bound (comparable to the one for Bayes mixtures) for certain important model classes. We discuss the application to Machine Learning tasks such as classification and hypothesis testing, and generalization to countable classes of i.i.d. models.
 Barron, A. R., Cover, T. M. (1991) Minimum complexity density estimation. IEEE Trans. on Information Theory 37: pp. 10341054 CrossRef
 Barron, A. R., Rissanen, J. J., Yu, B. (1998) The minimum description length principle in coding and modeling. IEEE Trans. on Information Theory 44: pp. 27432760 CrossRef
 Clarke, B. S., Barron, A.R. (1990) Informationtheoretic asymptotics of Bayes methods. IEEE Trans. on Information Theory 36: pp. 453471 CrossRef
 Gács, P. (1983) On the relation between descriptional complexity and algorithmic probability. Theoretical Computer Science 22: pp. 7193 CrossRef
 Grünwald P. and Langford J. 2004. Suboptimal behaviour of Bayes and MDL in classification under misspecification. In 17th Annual Conference on Learning Theory (COLT, pp. 331–347.
 Hutter M. 2001. Convergence and error bounds for universal prediction of nonbinary sequences. Proc. 12th Eurpean Conference on Machine Learning (ECML2001), pp. 239–250
 Hutter, M. (2003a) Convergence and loss bounds for Bayesian sequence prediction. IEEE Trans. on Information Theory 49: pp. 20612067 CrossRef
 Hutter, M. (2003b) Optimality of universal Bayesian prediction for general loss and alphabet. Journal of Machine Learning Research 4: pp. 9711000 CrossRef
 Hutter. M. 2003c. Sequence prediction based on monotone complexity. In Proc. 16th Annual Conference on Learning Theory (COLT2003), Lecture Notes in Artificial Intelligence, Berlin, Springer, pp. 506–521.
 Hutter, M. (2005) Sequential predictions based on algorithmic complexity. Journal of Computer and System Sciences 72: pp. 95117 CrossRef
 Levin, L. A. (1973) On the notion of a random sequence. Soviet Math. Dokl 14: pp. 14131416
 Li J. Q. 1999. Estimation of Mixture Models. PhD thesis, Dept. of Statistics. Yale University.
 Li M. and Vit’anyi P. M. B. 1997. An introduction to Kolmogorov complexity and its applications. Springer, 2nd edition.
 Poland J. and Hutter M. 2004a. Convergence of discrete MDL for sequential prediction. In 17th Annual Conference on Learning Theory (COLT), pp. 300–314.
 Poland J. and Hutter M. 2004b. On the convergence speed of MDL predictions for Bernoulli sequences. In International Conference on Algorithmic Learning Theory (ALT), pp. 294– 308.
 Poland J. and Hutter M. 2005. Strong asymptotic assertions for discrete MDL in regression and classification. In Benelearn 2005 (Ann. Machine Learning Conf. of Belgium and the Netherlands)
 Rissanen, J. J. (1996) Fisher Information and Stochastic Complexity. IEEE Trans. on Information Theory 42: pp. 4047 CrossRef
 Rissanen, J. J. (1999) Hypothesis selection and testing by the MDL principle. The Computer Journal 42: pp. 260269 CrossRef
 Solomonoff, R. J. (1978) Complexitybased induction systems: comparisons and convergence theorems. IEEE Trans. Information Theory IT24: pp. 422432 CrossRef
 Vitányi, P. M., Li, M. (2000) Minimum description length induction, Bayesianism, and Kolmogorov complexity. IEEE Trans. on Information Theory 46: pp. 446464 CrossRef
 Vovk, V. G. (1997) Learning about the parameter of the Bernoulli model. Journal of Computer and System Sciences 55: pp. 96104 CrossRef
 Zhang T. 2004. On the convergence of MDL density estimation. In Proc. 17th Annual Conference on Learning Theory (COLT), pp. 315–330,
 Zvonkin, A. K., Levin, L. A. (1970) The complexity of finite objects and the development of the concepts of information and randomness by means of the theory of algorithms. Russian Mathematical Surveys 25: pp. 83124 CrossRef
 Title
 MDL convergence speed for Bernoulli sequences
 Journal

Statistics and Computing
Volume 16, Issue 2 , pp 161175
 Cover Date
 20060601
 DOI
 10.1007/s1122200667463
 Print ISSN
 09603174
 Online ISSN
 15731375
 Publisher
 Kluwer Academic Publishers
 Additional Links
 Topics
 Industry Sectors
 Authors

 Jan Poland ^{(1)}
 Marcus Hutter ^{(2)}
 Author Affiliations

 1. Graduate School of Information Science and Technology, Hokkaido University, Japan
 2. IDSIA, Galleria 2, CH6928, Manno (Lugano), Switzerland