The New Palgrave Dictionary of Economics

Living Edition
| Editors: Palgrave Macmillan

Dynamic Programming and Markov Decision Processes

  • Steven A. Lippman
Living reference work entry


A great many problems in economics can be reduced to determining the maximum of a given function. Dynamic programming is one of a number of mathematical optimization techniques applicable in such problems. As will be illustrated, the dynamic programming technique or viewpoint is particularly useful in complex optimization problems with many variables in which time plays a crucial role. Unlike calculus-based techniques it does not require the function being optimized to be differentiable in the (decision) variables.


Dynamic Programming Optimal Policy Markov Decision Process Labour Income Constant Relative Risk Aversion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.


  1. Bellman, R. 1957. Dynamic programming. Princeton: Princeton University Press.Google Scholar
  2. Blackwell, D. 1962. Discrete dynamic programming. Annals of Mathematics and Statistics 33: 719–726.CrossRefGoogle Scholar
  3. Blackwell, D. 1965. Discounted dynamic programming. Annals of Mathematics and Statistics 36: 226–235.CrossRefGoogle Scholar
  4. Denardo, E.V. 1967. Contraction mappings in the theory underlying dynamic programming. SIAM Review 9: 165–177.CrossRefGoogle Scholar
  5. Denardo, E.V. 1982. Dynamic programming. Englewood Cliffs: Prentice-Hall.Google Scholar
  6. Hakansson, N.H. 1970. Optimal investment and consumption strategies under risk for a class of utility functions. Econometrica 38: 587–607.CrossRefGoogle Scholar
  7. Heyman, D., and M. Sobel. 1984. Stochastic models in operations research, vol. II. New York: McGraw-Hill.Google Scholar
  8. Howard, R.A. 1960. Dynamic programming and Markov processes. Cambridge, MA: MIT Press.Google Scholar
  9. Levhari, D., and L.J. Mirman. 1977. Savings and uncertainty with an uncertain horizon. Journal of Political Economy 85: 265–281.CrossRefGoogle Scholar
  10. Lippman, S. 1975. On dynamic programming with unbounded rewards. Management Science 21: 1225–1233.CrossRefGoogle Scholar
  11. Miller, B.L. 1974. Optimal consumption with a stochastic income stream. Econometrica 42: 253–266.CrossRefGoogle Scholar
  12. Miller, B.L. 1976. The effect on optimal consumption of increased uncertainty in labor income in the multi period case. Journal of Economic Theory 13: 154–167.CrossRefGoogle Scholar
  13. Mirman, L.J. 1971. Uncertainty and optimal consumption decisions. Econometrica 39: 179–185.CrossRefGoogle Scholar
  14. Phelps, E.S. 1962. The accumulation of risky capital: A sequential utility analysis. Econometrica 30: 729–743.CrossRefGoogle Scholar
  15. Ross, S. 1983. Introduction to stochastic dynamic programming. New York: Academic Press.Google Scholar

Copyright information

© The Author(s) 1987

Authors and Affiliations

  • Steven A. Lippman
    • 1
  1. 1.