Bayesian Inductive Inference and Maximum Entropy
The principles of Bayesian reasoning are reviewed and applied to problems of inference from data sampled from Poisson, Gaussian and Cauchy distributions. Probability distributions (priors and likelihoods) are assigned in appropriate hypothesis spaces using the Maximum Entropy Principle, and then manipulated via Bayes’ Theorem. Bayesian hypothesis testing requires careful consideration of the prior ranges of any parameters involved, and this leads to a quantitive statement of Occam’s Razor. As an example of this general principle we offer a solution to an important problem in regression analysis; determining the optimal number of parameters to use when fitting graphical data with a set of basis functions.
KeywordsPosterior Distribution Scale Parameter BAYESIAN Inference Maximum Entropy Bayesian Method
Unable to display preview. Download preview PDF.
- Gull, S.F. & Skilling, J. (1984). Maximum entropy method in image processing. IEE Proc., 131(F), 646–659.Google Scholar
- Jaynes, E.T. (1968). Prior probabilities. Reprinted in E.T. Jaynes: Papers on Probability, Statistics and Statistical Physics, ed. R. Rosenkrantz, 1983 Dordrecht: Reidel.Google Scholar
- Jaynes, E.T. (1976). Confidence intervals versus Bayesian Intervals. Reprinted in E.T. Jaynes: Papers on Probability, Statistics and Statistical Physics, ed. R. Rosenkrantz, 1983. Dordrecht: Reidel.Google Scholar
- Jaynes, E.T. (1986). Bayesian Methods — an Introductory Tutorial. In Maximum Entropy and Bayesian Methods in Applied Statistics. ed. J.H. Justice. Cambridge University Press.Google Scholar
- Jeffreys, H. (1939). Theory of Probability, Oxford University Press. Later editions 1948, 1961,1983.Google Scholar
- Shore, J.E. & Johnson, R.W. (1980). Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Trans. Info. Theory, IT-26, 26–39 and IT-29, 942–943.Google Scholar
- Skilling, J. (1986). The Axioms of Maximum Entropy. Presented at 1986 Maximum Entropy conference, Seattle, Washington (this volume).Google Scholar