A Survey of Convergence Results for Maximum Entropy Methods

  • J. M. Borwein
  • A. S. Lewis
Part of the Fundamental Theories of Physics book series (FTPH, volume 53)


Maximum entropy methods seek to estimate an unknown density function, typically nonnegative, on the basis of some known moments — integrals of the function with respect to given weights. The estimate is chosen to minimize some measure of entropy, a convex integral functional of the density, subject to the given moment constraints. A desirable feature of such a method is that the estimates should converge to the unknown density as the number of known moments grows. We survey recent results demonstrating how various types of convergence (weak-star, weak and norm in L 1 and L p , measure, and uniform) result from various properties of the entropy (strict convexity, smoothness, and growth conditions). This investigation may be seen as an extension of the classical study of the convergence of Fourier series to the framework of maximum entropy problems. Rather than present the most general theorems known, the unified pattern of the results is illustrated on a single fairly general model problem. References are given for the more general results and their proofs.


Maximum Entropy Constraint Qualification Maximum Entropy Method Strict Convexity Entropy Estimate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    D. Amir and Z. Ziegler. Polynomials of extremal LP-norm on the L,,-Unit sphere. Jounal of Approximation Theory,18:86-98, 1976.Google Scholar
  2. [2]
    E. Asplund. Fréchet differentiability of convex functions. Acta Mathematica, 121: 31 - 47, 1968.MathSciNetzbMATHCrossRefGoogle Scholar
  3. [3]
    J.M. Borwein and A.S. Lewis. Convergence in measure, mean and max norm for sequences in L1. Forthcoming.Google Scholar
  4. [4]
    J.M. Borwein and A.S. Lewis. Convergence of best entropy estimates. SIAM Journal on Optimization, 1: 191 - 205, 1991.MathSciNetzbMATHCrossRefGoogle Scholar
  5. [5]
    J.M. Borwein and A.S. Lewis. Duality relationships for entropy-like minimization problems. SIAM Journal on Control and Optimization, 29: 325 - 338, 1991.MathSciNetzbMATHCrossRefGoogle Scholar
  6. [6]
    J.M. Borwein and A.S. Lewis. On the convergence of moment problems. Transactions of the American Mathematical Society, 325: 249 - 271, 1991.MathSciNetzbMATHCrossRefGoogle Scholar
  7. [7l.
    J.M. Borwein and A.S. Lewis. Partially finite convex programming, Part I, Duality theory. Mathematical Programming B, pages 15 - 48, 1992.Google Scholar
  8. [8]
    J.M. Borwein and A.S. Lewis. Partially finite convex programming, Part II, Explicit lattice models. Mathematical Programming B, pages 49 - 84, 1992.Google Scholar
  9. [9]
    J.M. Borwein and A.S. Lewis. Partially-finite programming in L 1 and the existence of maximum entropy estimates. SIAM Journal on Optimization,1992. To appear. CORR 91-05, University of Waterloo.Google Scholar
  10. [10]
    J.M. Borwein and A.S. Lewis. Strong convexity and optimization. SIAM Journal on Optimization, 1993. To appear.Google Scholar
  11. [11]
    J.M. Borwein, A.S. Lewis, and M.A. Limber. Entropy minimization with lattice bounds. Technical Report CORR 92-05, University of Waterloo, 1992. Submitted to Journal of Approximation Theory.Google Scholar
  12. [12]
    J.M. Borwein and H. Wolkowicz. A simple constraint qualification in infinite dimensional programming. Mathematical Programming, 35: 83 - 96, 1986.MathSciNetzbMATHCrossRefGoogle Scholar
  13. [13]
    Peter Borwein and A.S. Lewis. Moment-matching and best entropy estimation. Technical Report CORR 91-03, University of Waterloo, 1991. Submitted to Journal of Mathematical Analysis and Applications.Google Scholar
  14. [14]
    B. Buck and V.A. Macaulay, editors. Maximum entropy in action. Oxford University Press, Oxford, 1991.Google Scholar
  15. [15]
    J.P. Burg. Maximum entropy spectral analysis. Paper presented at 37th meeting of the Society of Exploration Geophysicists, Oklahoma City, 1967.Google Scholar
  16. [16]
    E.W. Cheney. Introduction to approximation theory. McGraw-Hill, New York, 1966.zbMATHGoogle Scholar
  17. [17]
    D. Dacunha-Castelle and F. Gamboa. Maximum dentropie et problème des moments. Annales de lInstitut Henri Poincaré, 26: 567 - 596, 1990.MathSciNetzbMATHGoogle Scholar
  18. [18]
    J.E. Dennis and R.B. Schnabel. Numerical methods for unconstrained optimization and nonlinear equations. Prentice-Hall, New Jersey, 1983.zbMATHGoogle Scholar
  19. [19]
    J. Diestel. Sequences and Series in Banach Spaces. Springer-Verlag, New York, 1984.CrossRefGoogle Scholar
  20. [20]
    N. Dunford and J.T. Schwartz. Linear operators, volume 1. Interscience, New York, 1958.zbMATHGoogle Scholar
  21. [21]
    B. Forte, W. Hughes, and Z. Pales. Maximum entropy estimators and the problem of moments. Rendiconti di Matematica, Serie VII, 9: 689 - 699, 1989.MathSciNetzbMATHGoogle Scholar
  22. [22]
    F. Gamboa. Methode du Maximum dEntropie sur la Moyenne et Applications. PhD thesis, Universite Paris Sud, Centre dOrsay, 1989.Google Scholar
  23. [23]
    E. Gassiat. Probléme sommatoire par maximum dentropie. Comptes Rendues de lAcadémie des Sciences de Paris, Série I, 303: 675 - 680, 1986.MathSciNetzbMATHGoogle Scholar
  24. [24]
    R.K. Goodrich and A. Steinhardt. L2 spectral estimation. SIAM Journal on Applied Mathematics, 46: 417 - 428, 1986.MathSciNetzbMATHCrossRefGoogle Scholar
  25. [25]
    R.B. Holmes. Geometric functional analysis and its applications. Springer-Verlag, New York, 1975.zbMATHCrossRefGoogle Scholar
  26. [26]
    E.T. Jaynes. Prior probabilites. IEEE Transactions, SSC-4: 227 - 241, 1968.Google Scholar
  27. [27]
    A.S. Lewis. Pseudo-Haar functions and partially-finite programming. Manuscript.Google Scholar
  28. [28]
    A.S. Lewis. The convergence of entropic estimates for moment problems. In S. Fitzpatrick and J. Giles, editors, Workshop/Miniconference on Functional Analysis/Optimization, pages 100 - 115, Canberra, 1989. Centre for Mathematical Analysis, Australian National University.Google Scholar
  29. [29]
    A.S. Lewis. The convergence of Burg and other entropy estimates. Technical Report CORR 92-08, University of Waterloo, 1992.Google Scholar
  30. [30]
    A.S. Lewis. Facial reduction in partially finite convex programming. Technical Report CORR 92-07, University of Waterloo, 1992.Google Scholar
  31. [31]
    L.R. Mead and N. Papanicolaou. Maximum entropy in the problem of moments. Journal of Mathematical Physics, 25: 2404 - 2417, 1984.MathSciNetCrossRefGoogle Scholar
  32. [32]
    J. Navaza. The use of non-local constraints in maximum-entropy electron density reconstruction. Acta Crystallographica, A42: 212 - 223, 1986.Google Scholar
  33. [33]
    R. Nityananda and R. Narayan. Maximum entropy image reconstruction — a practical non-information-theoretic approach. Journal of Astrophysics and Astronomy, 3: 419450, 1982.Google Scholar
  34. [34]
    R.T. Rockafellar. Integrals which are convex functionals. Pacific Journal of Mathematics, 24: 525 - 539, 1968.MathSciNetzbMATHCrossRefGoogle Scholar
  35. [35]
    R.T. Rockafellar. Convex Analysis. Princeton University Press, Princeton, N.J., 1970.zbMATHGoogle Scholar
  36. [36]
    W. Rudin. Real and complex analysis. McGraw-Hill, New York, 1966.zbMATHGoogle Scholar
  37. [37]
    J. Skilling and S.F. Gull. The entropy of an image. SIAM-AMS Proceedings, 14: 167189, 1984.Google Scholar
  38. [38]
    M. Teboulle and I. Vajda. Convergence of best 0-entropy estimates. IEEE Transactions on Information Theory, 1992. To appear.Google Scholar
  39. [39]
    A. Visintin. Strong convergence results related to strict convexity. Communications on partial differential equations, 9: 439 - 466, 1984.MathSciNetzbMATHCrossRefGoogle Scholar
  40. [40]
    Z. Ziegler. Minimizing the 4,,,o-distortion of trigonometric polynomials. Journal of Mathematical Analysis and Applications, 61: 426 - 431, 1977.MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 1993

Authors and Affiliations

  • J. M. Borwein
    • 1
  • A. S. Lewis
    • 1
  1. 1.Department of Combinatorics and OptimizationUniversity of WaterlooWaterlooCanada

Personalised recommendations