Minimum Description Length Principle for Fat-Tailed Distributions

Abstract

While in theory many processes should have normal distribution or at least show asymptotic normality, in practice nonlinear systems often exhibit fat-tail distributions. Non-normality is hard to model, but also hard to detect (model selection). The problem is exacerbated by the varying complexity of the models, i.e. their propensity to overfit.

The Minimum Description Length principle applies Shannon’s information theory in statistical enquiry to balance between goodness of fit and model complexity. More specifically, the Normalized Maximum Likelihood (NML) model, stochastic distribution complexity are discussed.

Prior research has shown the distribution complexity for spherical distributions (uncorrelated identically distributed samples) in closed form.

The purpose of this paper is to extend the MDL framework to cover the independent samples case. A general and optimized numerical method for the calculation of the distribution complexity and the stochastic complexity is presented, with results shown for the Student-T distribution.