The MDL Principle

Abstract

We have mentioned the MDL principle on several occasions somewhat loosely as the principle that calls for finding the model and model class with which the data together with the model and model class, respectively, can be encoded with the shortest code length. Actually to apply the principle we must distinguish between two types of models — those for data compression and others for general statistical purposes such as prediction. In data compression, we apply the models to the same data from which the models are determined. Hence these models need not have any predictive power; and, in fact, to get the shortest code length we do not even need to fit models in the class considered, say, $\mathcal{M}_\gamma$ _γ. This is because the universal NML model gives a code length, which we called the stochastic complexity and which we consider to be the shortest for all intents and purposes.