Machine Learning

, Volume 66, Issue 2, pp 119-149

First online:

Open Access This content is freely available online to anyone, anywhere at any time.

Suboptimal behavior of Bayes and MDL in classification under misspecification

  • Peter GrünwaldAffiliated withCWI Email author 
  • , John LangfordAffiliated withYahoo Research


We show that forms of Bayesian and MDL inference that are often applied to classification problems can be inconsistent. This means that there exists a learning problem such that for all amounts of data the generalization errors of the MDL classifier and the Bayes classifier relative to the Bayesian posterior both remain bounded away from the smallest achievable generalization error. From a Bayesian point of view, the result can be reinterpreted as saying that Bayesian inference can be inconsistent under misspecification, even for countably infinite models. We extensively discuss the result from both a Bayesian and an MDL perspective.


Bayesian statistics Minimum description length Classification Consistency Inconsistency Misspecification