Maximum Entropy and Minimum Cross-Entropy Principles: Need for a Broader Perspective
Jaynes’ Maximum Entropy Principle (MaxEnt) has served as a unifying principle in the study of a wide variety of probabilistic systems transcending all disciplinary boundaries. Here, we are concerned about the two inverse problems of determining the most unbiased moment constraints, and the most unbiased entropy measure, when the remaining two probabilistic entities are specified. The need for these inverse problems arises in application areas and has the same relevance as MaxEnt. The problem of determining the most unbiased measure of entropy, however, takes us out of the framework of MaxEnt, where considerations of Generalized Measures of Entropy become essential. Departure from the well-established use of the Shannon measure has raised a variety of objections ranging all the way from the meaning of entropy to the mutilation of its uniqueness properties. The Generalized Maximum Entropy Principle(GMEP), which is the main contribution of this paper, takes cognizance of these criticisms by giving justifications for the use of generalized measures. Acceptance of this new model will result in extending the scope of MaxEnt to tackle problems which are at present beyond its scope.
A very similar argument is advanced in favour of a Generalized Minimum Cross-Entropy principle where other than Kullback-Leibler measure are considered.
KeywordsInverse Problem Maximum Entropy Generalize Entropy Shannon Entropy Entropy Measure
Unable to display preview. Download preview PDF.
- 4.Kapur J.N., “Measures of uncertainty, mathematical programming and physics”, Jour. Ind. Soc. Agri. Stat., Vol. 24, pp. 47–66, (1972).Google Scholar
- 8.Shore J. E., “Derivation of Equilibrium and Time-dependent Solution for Queuing Systems using Entropy Maximization”, Nat. Computer. Cong., Vol. 47, pp. 483–487, (1978).Google Scholar
- 10.Wilson A.G., Entropy in Urban and Regional Modelling, Pion, London, (1970).Google Scholar
- 11.Kesavan, H.K. and J.N.Kapur, “The Generalized Maximum Entropy Principle”, IEEE SMC Vol. 19, no.5, Sept/Oct 1989.Google Scholar
- 12.Kesavan, H.K. and J.N.Kapur, “On the Families of Solutions to Generalized Maximum Entropy and Minimum Cross-Entropy Problems,” Int. J. General Systems, 1989.Google Scholar