Skip to main content
SpringerLink
Log in
Book cover

Entropy Optimization and Mathematical Programming pp 51–124Cite as

Entropy Optimization Methods: Linear Case

  • Chapter

  • 279 Accesses

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 8))

Abstract

Let x ≡ (x 1,…,x n )T ≥ 0 be a nonnegative n-dimensional column vector and p ≡ (p 1,…,p n )T > 0 be a positive n-dimensional column vector. With the convention of 0 ln 0 = 0, we define the quantity \(\sum _{j = 1}^n{x_j}\ln ({x_j}/{p_j})\) to be the cross-entropy of x with respect to p, in a general sense. Note that when x and p are both probability distributions, i.e., \(\sum _{j = 1}^n{x_j} = \sum _{j = 1}^n{p_j} = 1\) this quantity becomes the commonly defined cross-entropy between the two probability distributions (see Chapter 1).

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Agmon, N., Alhassid, Y., and Levine, R.D., “An Algorithm for Determining the Lagrange Parameters in the Maximal Entropy formalism,” The Maximum Entropy Formalism, edited by R. D. Levine and M. Tribus, MIT Press, Cambridge, Massachusetts, 1979, pp. 207–209.

    Google Scholar 

  2. Anas, A., “Discrete Choice Theory, Information Theory and the Multinomial Logit and Gravity Models,” Transportation Research Part B, Vol. 17B, 1983, pp. 13–23.

    Article  Google Scholar 

  3. Bazaraa, M. S., and Shetty, C. M., Nonlinear Programming: Theory and Algorithms, John Wiley, New York, 1979.

    Google Scholar 

  4. Bellman, R.E., Mathematical Methods in Medicine, World Scientific, Singapore, 1983.

    Google Scholar 

  5. Ben-Tal A., and Charnes, A., “A Dual Optimization Framework for Some Problems of Information Theory and Statistics,” Problems of Control and Information Theory, Vol. 8, 1979, pp. 387–401.

    Google Scholar 

  6. Ben-Tal, A., Melman, A., and Zowe, J., “Curved Search Methods for Unconstrained Optimization,” Optimization, Vol. 21, 1990, pp. 669–695.

    Article  Google Scholar 

  7. Ben-Tal, A., Teboulle, M., and Charnes, A., “The Role of Duality in Optimization Problems involving Entropy Functionals with Applications to Information Theory,” Journal of Optimization Theory and Applications, Vol. 58, 1988, pp. 209–223.

    Article  Google Scholar 

  8. Bishop, Y.M.M., “Full Contingency Tables, Logits, and Split Contingency Tables,” Biometrika, Vol. 25, 1969, pp. 339–383.

    Google Scholar 

  9. Borwein, J.M., and Lewis, A.S., “Duality Relationships for Entropy-Like Minimization Problems,” SIAM Journal on Control and Optimization, Vol. 29, 1991, pp. 325–338.

    Article  Google Scholar 

  10. Borwein, J.M., and Lewis, A.S., “Partially-Finite Programming in L 1 and the Existence of Maximum Entropy Estimates,” SIAM Journal on Optimization, Vol. 3, 1993, pp. 248–267.

    Article  Google Scholar 

  11. Bregman, L.M., “The Relaxation Method of Finding the Common Point of Convex Sets and Its Application to the Solution of Problems in Convex Programming,” U.S.S.R. Computational Mathematics and Mathematical Physics, Vol. 7, 1967, pp. 200–217.

    Article  Google Scholar 

  12. Censor, Y., “Row-Action Methods for Huge and Sparse Systems and Their Applications,” SIAM Review, Vol. 23, 1981, pp. 444–466.

    Article  Google Scholar 

  13. Censor, Y., “On Linearly Constrained Entropy Maximization,” Linear Algebra and Its Applications, Vol. 80, 1986, pp. 191–195.

    Google Scholar 

  14. Censor, Y., Elfving, T., and Herman, G.T., “Special Purpose Algorithms for Linearly Constrained Optimization,” Maximum-Entropy and Bayesian Spectral Analysis and Estimation Problems, edited by C.R. Smith and G.J. Erickson, D. Reidel Publishing Company, 1987, pp. 241–254.

    Google Scholar 

  15. Censor, Y., and Lent, A., “An Iterative Row-Action Method for Interval Convex Programming,” Journal of Optimization Theory and Applications, Vol. 34, 1981, pp. 321–353.

    Article  Google Scholar 

  16. Censor, Y., De Pierro, A.R., Elfving, T., Herman, G.T., and Iusem, A.N., “On Iterative Methods for Linearly Constrained Entropy Maximization,” Banach Center Publications, Vol. 24, 1990, pp. 145–163.

    Google Scholar 

  17. Darroch, J.N., and Ratcliff, D., “Generalized Iterative Scaling for Log-linear Models,” Annals of Mathematical Statistics, Vol. 43, 1972, pp. 1470–1480.

    Article  Google Scholar 

  18. Dennis, J.E., and Schnabel, R.B., Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice Hall, Englewood Cliffs, New Jersey, 1983.

    Google Scholar 

  19. Dinkel, J.J., and Kochenberger, G.A., “Constrained Entropy Models: Solvability and Sensitivity,” Management Science, Vol. 25, 1979, pp. 555–564.

    Article  Google Scholar 

  20. Dinkel, J.J., Kochenberger, G.A., and Wong, S.N., “Entropy Optimization and Geometric Programming,” Environment and Planning, Vol. 9, 1977, pp. 419–427.

    Article  Google Scholar 

  21. Dinkel, J.J., and Wong, D.S., “External Zones in Trip Distribution Models: Characterization and Solvability,” Transportation Science, Vol. 18, 1984, pp. 253–266.

    Article  Google Scholar 

  22. Elfving, T., “On Some Methods for Entropy Maximization and Matrix Scaling,” Linear Algebra and its Applications, Vol. 34, 1980, pp. 321–339.

    Article  Google Scholar 

  23. Eriksson, J.R., “A Note on Solution of Large Sparse Maximum Entropy Problems with Linear Equality Constraints,” Mathematical Programming, Vol. 18, 1980, pp. 146–154.

    Article  Google Scholar 

  24. Erlander, S., “Accessibility, Entropy and the Distribution and Assignment of Traffic,” Transportation Research, Vol. 11, 1977, pp. 149–153.

    Article  Google Scholar 

  25. Erlander, S., “Entropy in Linear Programming,” Mathematical Programming, Vol. 21, 1981, pp. 137–151.

    Article  Google Scholar 

  26. Fang, S.-C., “A New Unconstrained Convex Programming Approach to Linear Programming,” Zeitschrift fur Operations Research, Vol. 36, 1992, pp. 149–161.

    Google Scholar 

  27. Fang, S.-C., Peterson, E.L., and Rajasekera, J.R., “Minimum Cross-Entropy Analysis with Entropy-Type Constraints,” Journal of Computational and Applied Mathematics, Vol. 39, 1992, pp. 165–178.

    Article  Google Scholar 

  28. Fang, S.-C., and Puthenpura, S., Linear Optimization and Extensions: Theory and Algorithms, Prentice Hall, Englewood Cliffs, New Jersey, 1993.

    Google Scholar 

  29. Fang, S.-C., and Rajasekera, J.R., “Quadratically Constrained Minimum Cross-Entropy Analysis,” Mathematical Programming, Vol. 44, 1989, pp. 85–96.

    Article  Google Scholar 

  30. Fang, S.-C., and Tsao, H.-S.J., “Linear Programming with Entropic Perturbation,” Zeitschrift fur Operations Research, Vol. 37, 1993, pp. 171–186.

    Google Scholar 

  31. Fang, S.-C., and Tsao, H.-S.J., “A Quadratically Convergent Global Algorithm for the Linearly-Constrained Minimum Cross-Entropy Problem,” European Journal of Operational Research, Vol. 79, 1994, pp. 369–378.

    Article  Google Scholar 

  32. Fang, S.-C., and Wu, S.-Y., “An Inexact Approach to Solving Linear Semi-Infinite Programming Problems,” Optimization, Vol. 28, 1994, pp. 291–299.

    Article  Google Scholar 

  33. Fiacco, A.V., and Kortanek, K.O., Semi-Infinite Programming and Applications, Lecture Notes in Economics and Mathematical Systems No. 215, Springer-Verlag, New York, 1983.

    Book  Google Scholar 

  34. Gordon, R., Bender, R., and Herman, G.T., “Algebraic Reconstruction Techniques (ART) for Three Dimensional Electron Microscopy and X-ray Photography,” Journal of Theoretical Biology, Vol. 29, 1970, pp. 471–481.

    Article  Google Scholar 

  35. Grandy, W.T. Jr., and Schick, L.H., editors, Proceedings of the 10th International Workshop on Maximum Entropy and Bayesian Methods, Kluwer Academic Publishers, Dordrecht, The Netherlands, 1991.

    Google Scholar 

  36. Guiasu, S., Information Theory with Applications, McGraw-Hill, New York, 1977.

    Google Scholar 

  37. Guiasu, S., “Maximum Entropy Condition in Queueing Theory,” Journal of Operational Research Society, Vol. 37, 1986, pp. 293–301.

    Google Scholar 

  38. Hoshino, K., Kumar, V., Kumar, U., “On Optimizing of Generalized Iterative Scaling Method,” Working Paper No. 89-22, School of Business, Carleton University, Ontario, Canada, 1989.

    Google Scholar 

  39. Jan, G.W., and Fang, S.-C., “A New Variant of the Primal Affine Scaling Algorithm for Linear Programs,” Optimization, Vol. 22, 1991, pp. 681–715.

    Article  Google Scholar 

  40. Jaynes, E.T., “Information Theory and Statistical Mechanics II,” Physics Review, Vol. 108, 1957, pp. 171–190.

    Article  Google Scholar 

  41. Jefferson, T.R., and Scott, C.H., “The Analysis of Entropy Models with Equality and Inequality Constraints,” Transportation Research, Vol. 138, 1979, pp. 123–132.

    Google Scholar 

  42. Jornsten, K.O., and Lundgren, J.T., “An Entropy-Based Model Split Model,” Transportation Research Part B, Vol. 23B, 1989, pp. 345–359.

    Article  Google Scholar 

  43. Kapur, J.N., Maximum-Entropy Models in Science and Engineering, Wiley Eastern Ltd., New Delhi, 1989.

    Google Scholar 

  44. Kortanek, K.O., Potra, F., and Ye, Y., “On Some Efficient Interior Point Methods for Nonlinear Convex Programming,” Linear Algebra and Its Applications, Vol. 152, 1991, pp. 169–189.

    Article  Google Scholar 

  45. Kumar, V., Hoshino, K., and Kumar, U., “An Application of the Entropy Maximization Approach in Shopping Area Planning,” International Journal of General Systems, Vol. 16, 1989, pp. 25–42.

    Article  Google Scholar 

  46. Kullback, S., Information and Statistics, John Wiley, New York, 1959.

    Google Scholar 

  47. Kullback, S., and Libler, R.A., “On Information and Sufficiency,” Annals of Mathematical Statistics, Vol. 22, 1951, pp. 79–86.

    Article  Google Scholar 

  48. Lamond, B., and Stewart, N.F., “Bregman’s Balancing Method,” Transportation Research Part B, Vol. 15B, 1981, pp. 239–248.

    Article  Google Scholar 

  49. Lent, A., “Maximum Entropy and MART,” in Image Analysis and Evaluation, SPSE Conference Proceedings, edited by R. Shaw, Toronto, Canada, 1976, pp. 249–257.

    Google Scholar 

  50. Lent, A., “A Convergent Algorithm for Maximum Entropy Image Restoration with a Medical X-Ray Application,” in Image Analysis and Evaluation, edited by R. Shaw, Society of Photographic Scientists and Engineers (SPSE), Washington, D.C., 1977, pp. 249–257.

    Google Scholar 

  51. Luenberger, D.G., Linear and Nonlinear Programming, 2nd Edition, Addison-Wesley, Reading, Massachusetts, 1984.

    Google Scholar 

  52. Minoux, M., Mathematical Programming, Wiley-Interscience, New York, 1986.

    Google Scholar 

  53. Peterson, E.L., “Geometric Programming,” SIAM Review, Vol. 19, 1976, pp. 1–45.

    Article  Google Scholar 

  54. De Pierro, A.R. and Iusem, A.N., “A Relaxed version of Bregman’s Method for Convex Programming,” Journal of Optimization Theory and Applications, Vol. 51, 1986, pp. 421–440.

    Article  Google Scholar 

  55. Potra, F., and Ye, Y., “A Quadratically Convergent Polynomial Algorithm for Solving Entropy Optimization Problems,” SIAM Journal on Optimization, Vol. 3, 1993, pp. 843–860.

    Article  Google Scholar 

  56. Rockafellar, R.T., Convex Analysis, Princeton University Press, Princeton, New Jersey, 1970.

    Google Scholar 

  57. Safwat, K.N.A., and Magnanti, T.L., “A Combined Trip Generation, Trip Distribution, Modal Split, and Trip Assignment Model,” Transportation Science, Vol. 22, 1988, pp. 14–30.

    Article  Google Scholar 

  58. Shore, J.E., “Minimum Cross-entropy Spectral Analysis,” IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. ASSP-29, 1981, pp. 230–237.

    Article  Google Scholar 

  59. Sheu, R.L., and Fang, S.-C., “On the Generalized Path-Following Methods for Linear Programming,” Optimization, Vol. 30, 1994, pp. 235–249.

    Article  Google Scholar 

  60. Smith, C.R., and Grandy, W.T., editors, Maximum-Entropy and Bayesian Methods in Inverse Problems, D. Reidel Publishing Company, Dordrecht, Holland, 1985.

    Google Scholar 

  61. Todd, M.J., and Ye, Y., “A Centered Projective Algorithm for Linear Programming,” Mathematics of Operations Research, Vol. 15, 1990, pp. 508–529.

    Article  Google Scholar 

  62. Tomlin, J.A., “A Mathematical Programming Model for the Combined Distribution-Assignment of Traffic,” Transportation Science, Vol. 5, 1971, pp. 122–140.

    Article  Google Scholar 

  63. Tomlin, J.A., and Tomlin, S.G., “Traffic Distribution and Entropy,” Nature, Vol. 220, 1968, pp. 974–976.

    Article  Google Scholar 

  64. Tsao, H.-S.J., and Fang, S.-C., “Linear Programming with Inequality Constraints via Entropic Perturbation,” International Journal of Mathematics and Mathematical Sciences, Vol. 19, 1996, pp. 177–184.

    Article  Google Scholar 

  65. Tsao, H.-S.J., Fang, S.-C., and Lee, D.N., “On the Optimal Entropy Analysis,” European Journal of Operational Research, Vol. 59, 1992, pp. 324–329.

    Article  Google Scholar 

  66. Tsao, H.-S.J., Fang, S.-C., and Lee, D.N., “A Bayesian Interpretation of the Linearly-Constrained Cross-Entropy Minimization Problem,” Engineering Optimization, Vol. 22, 1993, pp. 65–75.

    Article  Google Scholar 

  67. Wilson, A.G., “A Statistical Theory of Spatial Distribution Models,” Transportation Research, Vol. 1, 1967, pp. 253–269.

    Article  Google Scholar 

  68. Wu, J.-S., and Chan, W.C., “Maximum Entropy Analysis of Multiple-Server Queueing Systems,” Journal of Operational Research Society, Vol. 40, 1989, pp. 815–826.

    Google Scholar 

  69. Zhang, J., and Brockett, P.L., “Quadratically Constrained Information Theoretic Analysis,” SIAM Journal on Applied Mathematics, Vol. 47, 1987, pp. 871–885.

    Article  Google Scholar 

  70. Zhu, J., and Ye, Y., “A Path-Following Algorithm for a Class of Convex Programming Problems,” Working Paper No. 90-14, College of Business Administration, The University of Iowa, Iowa City, Iowa, 1990.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. North Carolina State University, Raleigh, North Carolina, USA

    S.-C. Fang

  2. International University of Japan, Urasa, Niigata, Japan

    J. R. Rajasekera

  3. University of California, Berkeley, California, USA

    H.-S. J. Tsao

Authors
  1. S.-C. Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. R. Rajasekera
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. H.-S. J. Tsao
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 1997 Springer Science+Business Media New York

About this chapter

Cite this chapter

Fang, SC., Rajasekera, J.R., Tsao, HS.J. (1997). Entropy Optimization Methods: Linear Case. In: Entropy Optimization and Mathematical Programming. International Series in Operations Research & Management Science, vol 8. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-6131-6_3

Download citation

  • DOI: https://doi.org/10.1007/978-1-4615-6131-6_3

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4613-7810-5

  • Online ISBN: 978-1-4615-6131-6

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation