Minimum Description Length Principle

Roos, Teemu

doi:10.1007/978-1-4899-7687-1_894

Minimum Description Length Principle

Teemu Roos³

Reference work entry
First Online: 14 April 2017

131 Accesses

Abstract

The minimum description length (MDL) principle states that one should prefer the model that yields the shortest description of the data when the complexity of the model itself is also accounted for. MDL provides a versatile approach to statistical modeling. It is applicable to model selection and regularization. Modern versions of MDL lead to robust methods that are well suited for choosing an appropriate model complexity based on the data, thus extracting the maximum amount of information from the data without over-fitting. The modern versions of MDL go well beyond the familiar \(\frac{k} {2} \log n\) formula.

This is a preview of subscription content, access via your institution.

Chapter: USD 29.95; Price excludes VAT (USA)

eBook: USD 799.99; Price excludes VAT (USA)

Hardcover Book: USD 899.99; Price excludes VAT (USA)

Learn about institutional subscriptions

Recommended Reading

Good review articles on MDL include Barron et al. (1998); Hansen and Yu (2001). The textbook by Grünwald (2007) is a comprehensive and detailed reference covering developments until 2007 Grünwald (2007).

Barron A, Cover T (1991) Minimum complexity density estimation. IEEE Trans Inf Theory 37(4):1034–1054
MathSciNet MATH CrossRef Google Scholar
Barron A, Rissanen J, Yu B (1998) The minimum description length principle in coding and modeling. IEEE Trans Inf Theory 44:2734–2760
MathSciNet MATH CrossRef Google Scholar
Fayyad U, Irani K (1993) Multi-interval discretization of continuous-valued attributes for classification learning. In: Bajczy R (ed) Proceedings of the 13th International Joint Conference on Artificial Intelligence and Minimum Description Length Principle, Chambery. Morgan Kauffman
Google Scholar
Grünwald P (2007) The Minimum Description Length Principle. MIT Press, Cambridge
Google Scholar
Hansen M, Yu B (2001) Model selection and the principle of minimum description length. J Am Stat Assoc 96(454):746–774
MathSciNet MATH CrossRef Google Scholar
Lam W, Bacchus F (1994) Learning Bayesian belief networks: an approach based on the MDL principle. Comput Intell 10:269–293
CrossRef Google Scholar
Rissanen J (1978) Modeling by shortest data description. Automatica 14(5):465–658
MATH CrossRef Google Scholar
Rissanen J (1984) Universal coding, information, prediction, and estimation. IEEE Trans Inf Theory 30:629–636
MathSciNet MATH CrossRef Google Scholar
Rissanen J (1986) Stochastic complexity and modeling. Ann Stat 14(3):1080–1100
MathSciNet MATH CrossRef Google Scholar
Rissanen J (1996) Fisher information and stochasic complexity. IEEE Trans Inf Theory 42(1):40–47
MATH CrossRef Google Scholar
Rissanen J (2000) MDL denoising. IEEE Trans Inf Theory 46(7):2537–2543
MATH CrossRef Google Scholar
Schwarz G (1978) Estimating the dimension of a model. Ann Stat 6(2):461–464
MathSciNet MATH CrossRef Google Scholar
Silander T, Roos T, Myllymäki P (2010) Learning locally minimax optimal Bayesian networks. Int J Approx Reason 51(5):544–557
MathSciNet CrossRef Google Scholar
Speed T, Yu B (1993) Model selection and prediction: normal regression. Ann Inst Stat Math 45(1):35–54
MathSciNet MATH CrossRef Google Scholar
Wallace C, Boulton D (1968) An information measure for classification. Comput J 11(2):185–194
MATH CrossRef Google Scholar
Wei C (1992) On predictive least squares principles. Ann Stat 20(1):1–42
MathSciNet MATH CrossRef Google Scholar
Weinberger M, Rissanen J, Feder M (1995) A universal finite memory source. IEEE Trans Inf Theory 41(3):643–652
MATH CrossRef Google Scholar
Yuri Shtarkov (1987) Universal sequential coding of single messages. Probl Inf Transm 23(3):3–17
MathSciNet Google Scholar

Download references

Author information

Authors and Affiliations

Department of Computer Science, Helsinki Institute for Information Technology, University of Helsinki, Helsinki, Finland
Teemu Roos

Authors

Teemu Roos
View author publications
You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Teemu Roos .

Editor information

Editors and Affiliations

The University of New South Wales, Sydney, NSW, Australia
Claude Sammut
Faculty of Information Technology, Monash University, Melbourne, VIC, Australia
Geoffrey I. Webb

Rights and permissions

Reprints and Permissions

Copyright information

About this entry

Cite this entry

Roos, T. (2017). Minimum Description Length Principle. In: Sammut, C., Webb, G.I. (eds) Encyclopedia of Machine Learning and Data Mining. Springer, Boston, MA. https://doi.org/10.1007/978-1-4899-7687-1_894

Download citation

DOI: https://doi.org/10.1007/978-1-4899-7687-1_894
Published: 14 April 2017
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4899-7685-7
Online ISBN: 978-1-4899-7687-1
eBook Packages: Computer ScienceReference Module Computer Science and Engineering