On the Computation of Entropy Prior Complexity and Marginal Prior Distribution for the Bernoulli Model
As the size and complexity of models grow, the choice of the best model becomes a difficult and challenging task. Once the best model is specified, the goodness of fit of the model needs to be examined first. A highly complex model may provide a good fit, but giving no consideration to model complexity could result in incorrect estimates of parameter values and predictions. In order to improve the model selection process, model complexity needs to be defined clearly. This article studies different aspects of model complexity and discusses the extent to which they can be measured. The most common attribute that is usually ignored from many complexity measures is the parameter prior, which is an inherent part of the model and could impact the complexity significantly. The concept of parameter prior and its connection to model complexity are therefore discussed here, and some relationships to the entropy measure elements are also addressed.
KeywordsModel complexity Entropy Prior odds Posterior odds Marginal probability
AMS Subject Classification62K15 62-07 62J12
Unable to display preview. Download preview PDF.
- Berger, A. L, S. Della Pietra, and V. J. Della Pietra. 1996. A maximum-entropy approach to natural language processing. Comput. Linguistics, 22, 39–71.Google Scholar
- Catalan, R. G., J. Garay, and R. López-Ruiz. 2002. Features of the extension of a statistical measure of complexity for continuous systems. Phys. Rev. E, 66, 011102(6).Google Scholar
- Caticha, A. 2007. Information and entropy. In Bayesian inference and maximum entropy methods in science and engineering, ed. K. Knuth et al., AIP Conf. Proc., vol. 954, 11. New York, NY: AIP.Google Scholar
- Charles, S. B. 2002. A comparison of marginal likelihood computation methods. In COMPSTAT 2002: Proceedings in computational statistics, ed. W. Härdle and B. Ronz, 111–117. Berlin, Heidelberg: Springer-Verlag.Google Scholar
- Grünwald, P. D. 2005. MDL tutorial. In Advances in minimum description length: Theory and applications, ed. P. D. Grünwald, I. J. Myung, and M. A. Pitt, 16–17. Cambridge, MA: MIT Press.Google Scholar
- Grünwald, P. D. 2007. The minimum description length principle. Cambridge, MA: MIT Press.Google Scholar
- Rissanen, J. 2005. Complexity and information in modeling. Chapter IV In Computability, complexity and constructivity in economic analysis, ed. K. Velupillai, chap. IV. Oxford, UK: Blackwell.Google Scholar
- Vanpaemel, W. 2009. Measuring model complexity with the prior predictive. In Advances in neural information processing systems (NIPS), ed. Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, vol. 22, 1919–1927. Red Hook, NY: Curran Associates.Google Scholar
- Wallis, K. F. 2006. A note on the calculation of entropy from histograms. Unpublished paper, University of Warwick, Coventry, UK.Google Scholar