Machine Learning

, Volume 66, Issue 2, pp 119–149

Suboptimal behavior of Bayes and MDL in classification under misspecification

Open AccessArticle

DOI: 10.1007/s10994-007-0716-7

Cite this article as:
Grünwald, P. & Langford, J. Mach Learn (2007) 66: 119. doi:10.1007/s10994-007-0716-7


We show that forms of Bayesian and MDL inference that are often applied to classification problems can be inconsistent. This means that there exists a learning problem such that for all amounts of data the generalization errors of the MDL classifier and the Bayes classifier relative to the Bayesian posterior both remain bounded away from the smallest achievable generalization error. From a Bayesian point of view, the result can be reinterpreted as saying that Bayesian inference can be inconsistent under misspecification, even for countably infinite models. We extensively discuss the result from both a Bayesian and an MDL perspective.


Bayesian statisticsMinimum description lengthClassificationConsistencyInconsistencyMisspecification
Download to read the full article text

Copyright information

© Springer Science + Business Media, LLC 2007

Authors and Affiliations

  1. 1.CWIAmsterdamThe Netherlands
  2. 2.Yahoo ResearchNew York