Maximum Entropy and Bayesian Methods pp 213-222 | Cite as

# Foundations: Indifference, Independence & MaxEnt

## Abstract

Through completing an under specified probability model, Maximum Entropy (MaxEnt) supports non-monotonic inferences. Some major aspects of how this is done by MaxEnt can be understood from the background of two principles of rational decision: the concept of Indifference and the concept of Independence. In a formal specification MaxEnt can be viewed as (conservative) extension of these principles; so these principles shed light on the “magical” decisions of MaxEnt. But the other direction is true as well: Since MaxEnt is a “correct” representation of the set of models (Concentration Theorem), it elucidates these two principles (e.g. it can be shown, that the knowledge of independences can be of very different information-theoretic value). These principles and their calculi are not just arbitrary ideas: When extended to work with qualitative constraints which are modelled by probability intervals, each calculus can be successfully applied to V.Lifschitz’s Benchmarks of Non-Monotonic Reasoning and is able to infer some instances of them ([Lifschitz88]). Since MaxEnt is strictly stronger than the combination of the two principles, it yields a powerful tool for decisions in situations of incomplete knowledge. To give an example, a well-known problem of statistical inference (Simpson’s Paradox) will serve as an illustration throughout the paper.

## Keywords

Maximum Entropy Undirected Graph Linear Constraint Propositional Logic Elementary Event## Preview

Unable to display preview. Download preview PDF.

## References

- [Adams75]E.W. Adams, “The Logic of Conditionals”, D.Reidel Dordrecht Netherlands, 1975.MATHGoogle Scholar
- [Bacchus90]F. Bacchus, “Lp — A Logic for Statistical Information”,
*Uncertainty in Artificial Intelligence**5*, pp. 3–14, Elsevier Science, ed.: M. Henrion, R.D. Shachter, L.N. Kanal, J.F. Lemmer, 1990.Google Scholar - [Bacchus94]F. Bacchus, A.J. Grove, J.Y. Halpern, D. Koller, “From Statistical Knowledge Bases to Degrees of Belief”,
*Technical Report*(available via ftp at logos.uwaterloo. ca: /pub/bacchus), 1994.Google Scholar - [Blyth73]C. Blyth, “Simpson’s Paradox und mutually favourable Events”,
*Journal of the American Statistical Association*, Vol. 68, p. 746, 1973.MathSciNetMATHCrossRefGoogle Scholar - [Cheeseman88]P. Cheeseman, “An Inquiry into Computer Understanding”,
*Computational Intelligence*, Vol. 4, pp. 58 – 66, 1988.CrossRefGoogle Scholar - [Cox79]R.T. Cox, “Of Inference and Inquiry — An Essay in Inductive Logic”, in:
*The Maximum Entropy Formalism*, MIT Press, ed.: Levine & Tribus, pp. 119–167, 1979.Google Scholar - [Howson93]C. Howson, P. Urbach. Urbach,“Scientific Reasoning: The Bayesian Approach”, 2nd Edition, Open Court, 1993.Google Scholar
- [Jaynes78]E.T. Jaynes, “Where do we stand on Maximum Entropy?”, 1978, in: E.T. Jaynes:
*Papers on Probability, Statistics and Statistical Physics*, pp. 210 – 314, Kluwer Academic Publishers, ed.: R.D. Rosenkrantz, 1989.Google Scholar - [Jaynes82]E.T. Jaynes, “On the Rationale of Maximum-Entropy Methods”,
*Proceedings of the IEEE*, Vol. 70, No. 9, pp. 939 – 952, 1982.CrossRefGoogle Scholar - [Lewis59]P.M. Lewis, “Approximating Probability Distributions to Reduce Storage Requirements”,
*Information and Control**2*, pp. 214–225, 1959.MathSciNetMATHCrossRefGoogle Scholar - [Lifschitz88]V. Lifschitz, “Benchmark Problems for Formal nonmonotonic Reasoning”,
*Lecture Notes in Artificial Intelligence Non-Monotonie Reasoning*, Vol. 346, pp. 202–219, ed.: Reinfrank et al., 1988.Google Scholar - [Neapolitan90]R.E. Neapolitan, “Probabilistic Reasoning in Expert Systems: Theory and Algorithms”, John Wiley & Sons, 1990.Google Scholar
- [Paris89]J.B. Paris, A. Vencovska, “On the Applicability of Maximum Entropy to Inexact Reasoning”,
*Int. Journal of approximate reasoning*, Vol. 3, pp. 1 – 34, 1989.MathSciNetMATHCrossRefGoogle Scholar - [Paris90]J.B. Paris, A. Vencovska, “A note on the Inevitability of Maximum Entropy”,
*Int. Journal of approximate reasoning*, Vol. 4, pp. 183–223, 1990.MathSciNetMATHCrossRefGoogle Scholar - [Pearl88]J. Pearl, “Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference”, Kaufmann, San Mateo, CA, 1988.Google Scholar
- [Shore80]J.E. Shore, R.W. Johnson, “Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross Entropy”,
*IEEE Transactions on Information Theory*, Vol. IT-26, No. 1, pp. 26– 37, 1980.MathSciNetCrossRefGoogle Scholar - [Skilling88]J. Skilling, “The Axioms of Maximum Entropy, Maximum-Entropy and Bayesian Methods in Science and Engineering, Vol. 1 – Foundations”, Kluwer Academic, ed.: G.J. Erickson, C.R. Smith, Seattle Univ. Washington, 1988.Google Scholar