Classic Maximum Entropy

  • John Skilling
Part of the Fundamental Theories of Physics book series (FTPH, volume 36)


This paper presents a fully Bayesian derivation of maximum entropy image reconstruction. The argument repeatedly goes from the particular to the general, in that if there are general theories then they must apply to special cases. Two such special cases, formalised as the “Cox axioms “, lead to the well-known fact that Bayesian probability theory is the only consistent language of inference. Further cases, formalised as the axioms of maximum entropy, show that the prior probability distribution for any positive, additive distribution must be monotonic in the entropy. Finally, a quantified special case shows that this monotonic function must be the exponential, leaving only a single dimensional scaling factor to be determined a posteriori. Many types of distribution, including probability distributions themselves, are positive and additive, so the entropy exponential is very general.

The following paper (Gull 1989) applies these ideas to image reconstruction, showing how a sophisticated treatment can incorporate prior expectation of spatial correlations.


Maximum Entropy Maximum Entropy Method Testable Information Prior Probability Distribution Incoherent Light 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Cox, R.P. (1946). Probability, Frequency and Reasonable Expectation. Am. Jour. Phys. 17, 1–13.CrossRefGoogle Scholar
  2. Frieden, B.R. (1972). Restoring with maximum likelihood and maximum entropy. J. Opt. Soc. Am., 62, 511–518.CrossRefGoogle Scholar
  3. Gull, S.F. (1989). Developments in maximum entropy data analysis. In these Proceedings.Google Scholar
  4. Gull, S.F. & Daniell, G.J. (1979). The maximum entropy method. In Image Formation from Coherence Functions in Astronomy, ed. C. van Schooneveld, pp. 219–225, Reidel.CrossRefGoogle Scholar
  5. Gull, S.F. & Skilling, J. (1984). The maximum entropy method. In Indirect Imaging, ed. J.A. Roberts. Cambridge University Press.Google Scholar
  6. Jaynes, E.T. (1978). “Where do we stand on maximum entropy? Reprinted in E.T. Jaynes: Papers on Probability, Statistics and Statistical Physics, ed. R. Rosenkrantz, 1983 Dordrecht: Reidel.Google Scholar
  7. Jaynes, E.T. (1986). Monkeys, kangaroos and N. In Maximum Entropy and Bayesian Methods in Applied Statistics, ed. J.H. Justice, pp. 26–58, Cambridge Univ. Press.CrossRefGoogle Scholar
  8. Klir, G.J. (1987). Where do we stand on measures of uncertainty, ambiguity, fuzziness and the like? Fuzzy sets and systems, 24, 141–160.MathSciNetzbMATHCrossRefGoogle Scholar
  9. Levine, R.D. (1986). Geometry in classical statistical thermodynamics, J. Chem. Phys., 84, 910–916.MathSciNetCrossRefGoogle Scholar
  10. Rodriguez, C. (1989). The metrics induced by the Kullback number. In these Proceedings.Google Scholar
  11. Shore, J.E. & Johnson, R.W. (1980). Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Trans.Info.Theory, IT-26, 26–39MathSciNetCrossRefGoogle Scholar
  12. Shore, J.E. & Johnson, R.W. (1980). Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Trans.Info.Theory, IT-29 942–943.MathSciNetGoogle Scholar
  13. Skilling, J. (1988). The axioms of maximum entropy. In Maximum Entropy and Bayesian Methods in Science and Engineering, Vol. 1., ed. G.J. Erickson & C.R. Smith, pp. 173–188. Kluwer.CrossRefGoogle Scholar
  14. Smith, C.R. & Erickson, G.J. (1989). From rationality and consistency to Bayesian probability. In these Proceedings.Google Scholar
  15. Tikochinsky, Y., Tishby, N.Z. & Levine, R.D. (1984). Consistent inference of probabilities for reproducible experiments. Phys. Rev. Lett., 52, 1357–1360.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 1989

Authors and Affiliations

  • John Skilling
    • 1
  1. 1.Dept. of Applied Mathematics and Theoretical PhysicsCambridgeUK

Personalised recommendations