Some New Perspectives on Maximum Entropy Techniques in Water Resources Research

Abstract

After a brief expository account of the Shannon-Jaynes principle of maximum entropy (POME) for discrete and continuous variables, we give here an account of some recent research work which (i) a “histogram” method to contrast the discrete and continuous modes computation and the role of histogram in actual practice when dealing with continuous probability distributions. (ii) The idea of mean logarithmic decrement associated with a probability distribution is introduced and is shown to be related to the concept of differential entropy. The mean with respect to an arbitrary probability distribution of the logarithm of the ratio of the probability density function for an exponential distribution is discussed in the context of hydrological investigations. Unlike the entropy of a continuous probability distribution introduced in (i), this quantity which is an example of the Kullback-Leibler (KL) Information, is always positive and invariant under coordinate transformation. (iii) The constraints entering into POME as well as a minimum K L information are identified as a class of sufficient statistics which determine the unknown parameters in the probability density functions that occur in the most commonly used hydrological models. (iv) An example of (iii) where only the first two moments in a semi-infinite domain are givens is discussed to shed light on the limitations of POME, recentky recognized by Wragg and coworkers, and is made relevant to the work of Sonuga on rainfall-run off relationship. Finally, (v) a method of generating probability distributions starting from one basic distribution employing coordinate transformations is given. This in conjuction with (iii) leads to the notions of “Physical constraints” in contrast to the “mathematical constraints” in examing parameter estimation.