Skip to main content
SpringerLink
Log in

Water Reservoir Applications of Markov Decision Processes

  • Chapter
Handbook of Markov Decision Processes

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 40))

Abstract

Decision problems in water resources management are usually stochastic, dynamic and multidimensional. MDP models have been used since the early fifties for the planning and operation of reservoir systems because the natural water inflows can be modeled using Markovian stochastic processes and the transition equations of mass conservation for the reservoir storages are akin to those found in inventory theory. However, the “curse of dimensionality” has been a major obstacle to the numerical solution of MDP models for systems with several reservoirs. Also, the use of optimization models for the operation of multipurpose reservoir systems is not so widespread, due to the need for negotiations between different users, with dam operators often relying on operating rules obtained by simulation models.

In this chapter, we present the basic concepts of reservoir management and we give a brief survey of stochastic inflow models based on statistical hydrology. We also present a stochastic dynamic programming model for the planning and operation of a system of hydroelectric reservoirs, and we discuss some applications and computational issues. We feel many research opportunities exist both in the enhancement of computational methods and in the modeling of reservoir applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 299.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 379.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 379.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Archibald, T.W., Buchanan, C.S., McKinnon, K.I.M and Thomas, L.C. (1999) Nested Benders decomposition and dynamic programming for reservoir optimization. Journal of the Operational Research Society 50, pp. 468–479.

    Google Scholar 

  2. Archibald, T.W., McKinnon, K.I.M and Thomas, L.C. (1997). An aggregate stochastic dynamic programming model of multiple reservoir systems. Water Resources Research 33, pp. 333–340.

    Article  Google Scholar 

  3. Bertsekas, D.P. (1976). Dynamic Programming and Stochastic Control. Academic Press, New York.

    Google Scholar 

  4. Box, G.E.P. and Jenkins, G.M. (1976). Time Series Analysis: Forecasting and Control. Holden-Day, San Francisco.

    Google Scholar 

  5. Camacho, F., McLeod, A.I. and Hipel, W.H. (1985). Contemporaneous autoregressive-moving average (CARMA) modeling in water resources. Water Resources Bulletin 21, pp. 709–720.

    Article  Google Scholar 

  6. Chen, V.C.P., Ruppert, D. and Shoemaker, C.A. (1999). Applying experimental design and regression splines to high-dimensional continuous-state stochastic dynamic programming. Operations Research 47, pp. 38–53.

    Article  Google Scholar 

  7. Eschenbach, E.A., Shoemaker, C.A. and Caffey, H.M. (1995). Parallel algorithms for stochastic dynamic programming with continuous state and control variables. ORSA J. on Computing 7, pp. 386–401.

    Article  Google Scholar 

  8. Fletcher, S.G. and Ponnambalam, K. (1998). A constrained state formulation for the stochastic control of multi-reservoir systems. Water Resources Research 34, pp. 257–270.

    Article  Google Scholar 

  9. Foufoula-Georgiou, E. and Kitanidis, P.K., (1988). Gradient dynamic programming for stochastic optimal control of multidimensional water resources systems. Water Resources Research 24, pp. 1345–1359.

    Article  Google Scholar 

  10. Gal, S. (1979). Optimal management of a multi-reservoir water supply system. Water Resources Research 15, pp. 737–749.

    Article  Google Scholar 

  11. Georgakakos, A.P., (1989). Extended linear quadratic Gaussian control for the real time operation of reservoir systems. In Dynamic Programming for Optimal Water Resources Systems Analysis, Prentice Hall, Esogbue Editor, pp. 329–360.

    Google Scholar 

  12. Gessford, J. and Karlin, S. (1958). Optimal policy for hydroelectric operations. In Studies on the Mathematical Theory of Inventory and Production, Stanford University Press, Stanford, Calif., pp. 179–200.

    Google Scholar 

  13. Hanscom, M. A., Lafond, L., Lasdon, L. and Pronovost, G. (1980). Modelling and resolution of the medium term energy generation planning problem for a large hydro-electric system. Management Science 26, pp. 659–668.

    Article  Google Scholar 

  14. Hipel, K.W., McLeod, A.I. and Lennox, W.C. (1977). Advances in Box- Jenkins modeling 1: Model construction. Water Resources Research 13, pp. 567–575.

    Article  Google Scholar 

  15. Jacobs, J.J., Freeman, G., Grygier, J., Morton, D., Schultz, G.L., Staschus, K., and Stedinger, J.R. (1995). Socrates: a system for scheduling hydroelectric generation under uncertainty. Annals of Operations Research 59, pp. 99–133.

    Article  Google Scholar 

  16. Johannesen, A. and Flatabo, N. (1989). Scheduling methods in operation planning of hydro-dominated power production system. Electrical Power & Energy Systems 11 pp. 189–199.

    Article  Google Scholar 

  17. Johnson, S.A., Stedinger, J.R., Shoemaker C.A., Li, Y. and Tejada-Guibert, J.A. (1993). Numerical solution of continuous-state dynamic programs using linear and spline interpolation. Operations Research 41 pp. 484–500.

    Article  Google Scholar 

  18. Kelman, J., Stedinger, J.R., Cooper, L.A., Hsu, E. and Tuan, S. (1990). Sampling stochastic dynamic programming applied to reservoir operation. Water Resources Research 26, pp. 447–454.

    Article  Google Scholar 

  19. Kitanidis, P.K. and Foufoula-Georgiou, E. (1987). Error analysis of conventional discrete and gradient dynamic programming. Water Resources Research 23, pp. 845–856.

    Article  Google Scholar 

  20. Lamond, B. F. (1997) Stochastic reservoir optimization using piecewise polynomial approximations. Working paper, Faculté des sciences de l’administration, Université Laval, Québec, Canada.

    Google Scholar 

  21. Lamond, B. F. and Bachar, B. (1998). Une étude numérique de la discrétisation des apports aléatoires pour un réservoir non saisonnier. INFOR 36, pp. 247–260.

    Google Scholar 

  22. Lamond, B. F. and Boukhtouta, A. (1996). Optimizing long-term hydro-power production using Markov decision processes. International Transactions in Operational Research 3, pp. 223–241.

    Article  Google Scholar 

  23. Lamond, B. F. and Lang, P. (1996). Lower bounding aggregation and direct computation for an infinite horizon one-reservoir model. European Journal of Operational Research 95, pp. 404–410.

    Article  Google Scholar 

  24. Lamond, B.F. and Sobel, M.J. (1995). Exact and approximate solutions of affine reservoir models. Operations Research 43, pp. 771–780.

    Article  Google Scholar 

  25. Little, J.D.C. (1955). The use of storage water in a hydroelectric system. Operations Research 3, pp. 187–197.

    Article  Google Scholar 

  26. Loaiciga, H.A. and Marino, M.A. (1985) An approach to parameter estimation and stochastic control in water resources with an application to reservoir operation. Water Resources Research 21, pp. 1575–1584.

    Article  Google Scholar 

  27. Loucks, D. P. and Sigvaldason, O. T. (1982). Multiple-reservoir operation in North America. In Z. Kaczmarek and J. Kindler (Eds), The Operation of Multiple Reservoir Systems, pp. 2–103. II AS A, Laxenburg, Austria.

    Google Scholar 

  28. Maidment, D. R. (1993). Handbook of Hydrology. McGraw-Hill, New-York.

    Google Scholar 

  29. Massé, P. B. D. (1946). Les Réserves et la Régulation de l’Avenir dans la Vie Économique. Hermann, Paris.

    Google Scholar 

  30. McLaughlin, D. and Veiasco, H.L. (1990). Real-time control of a system of large hydropower reservoirs. Water Resources Research 26, pp. 623–635.

    Article  Google Scholar 

  31. McLeod, A.I., Hipel, K.W. and Lennox, W.C. (1977). Advances in Box- Jenkins modeling 2: Applications. Water Resources Research 13, pp. 577–586.

    Article  Google Scholar 

  32. Oliveira, R. and Loucks, D. P. (1997). Operating rules for multi-reservoir systems. Water Resources Research 33, pp. 839–852.

    Article  Google Scholar 

  33. Pereira, M.V.F. and Pinto, L.M.V.G. (1985). Stochastic optimization of a multi-reservoir hydroelectric system: A decomposition approach. Water Resources Research 21, pp. 779–792.

    Article  Google Scholar 

  34. Pereira, M.V.F. and Pinto, L.M.V.G. (1991). Multi-stage stochastic optimization applied to energy planning. Mathematical Programming 52, pp. 359–375.

    Article  Google Scholar 

  35. Read, E. G. (1989). A dual approach to stochastic programming for reservoir release scheduling. In Dynamic Programming for Optimal Water Resources Systems Analysis, Prentice Hall, Esogbue Editor, pp. 361–372.

    Google Scholar 

  36. Salas, J.D., Tabios III, G.Q. and Bartolini, P. (1985). Approaches to multivariate modeling of water resources time series. Water Resources Bulletin 21, pp. 683–708.

    Article  Google Scholar 

  37. Salinger, D.H. (1997). A Splitting Algorithm for Multistage Stochastic Programming with Application to Hydropower Scheduling. Ph.D. thesis, University of Washington.

    Google Scholar 

  38. Shaw, E. M. (1988). Hydrology in Practice. Van Nostrand Reinhold, England.

    Google Scholar 

  39. Soares, S. and Carneiro, A.A.F.M. (1990). Optimal operation of reservoirs for electric generation. Proceedings of the IEEE Power Engineering Society 1990 Summer Meeting, Minneapolis, Minnesota.

    Google Scholar 

  40. Stedinger, J.R., Lettenmaier, D.P. and Vogel, R.M. (1985). Multisite ARMA(1,1) and disaggregation models for annual streamflow generation. Water Resources Research 21, pp. 497–509.

    Article  Google Scholar 

  41. Stedinger, J.R., Suie, B.F. and Loucks, D.P. (1984). Stochastic dynamic programming models for reservoir operation optimization. Water Resources Research 20, pp. 1499–1505.

    Article  Google Scholar 

  42. Stedinger, J.R. and Vogel, R. M. (1984). Disaggregation procedures for generating serially correlated flow vectors. Water Resources Research 20, pp. 47–56.

    Article  Google Scholar 

  43. Tejada-Guibert, J.A., Johnson, S.A. and Stedinger, J.R, (1993). Comparison of two approaches for implementing multi-reservoir operating policies derived using stochastic dynamic programming. Water Resources Research 29, pp. 3969–3980.

    Article  Google Scholar 

  44. Tejada-Guibert, J.A., Johnson, S.A. and Stedinger, J.R. (1995). The value of hydrologie information in stochastic dynamic programming models of a multi-reservoir system. Water Resources Research 31, pp. 2571–2579.

    Article  Google Scholar 

  45. Terry, L. A., Pereira, M. V. F., Araripe Neto, T. A., Silva, L. F. C. A. and Sales, P. R. H. (1986). Coordinating the energy generation of the Brazilian national hydrothermal electrical generating system. Interfaces 16, pp. 16–38.

    Article  Google Scholar 

  46. Turgeon, A. (1980). Optimal operation of multi-reservoir power systems with stochastic inflows. Water Resources Research 16, pp. 275–283.

    Article  Google Scholar 

  47. Turgeon, A. (1981) A decomposition method for the long-term scheduling of reservoirs in series. Water Resources Research 17, pp. 1565–1570.

    Article  Google Scholar 

  48. Turgeon, A. (1992). PERESE: The New Hydro-Québec optimization model for the long-term scheduling of reservoirs. In Hydropower’92: Proceedings of the 2nd International Conference on Hydropower, Held in Lillehamer, Norway Ed. Broch, E. and Lysne, D.K., Balkema publisher, Rotterdam, pp. 603–612.

    Google Scholar 

  49. Turgeon, A. and Charbonneau, R. (1998). An aggregation - disaggregation approach to long-term reservoir management. Water Resources Research 34, pp. 3585–3594.

    Article  Google Scholar 

  50. Valencia, R.D. and Schaake, J.C. (1973). Disaggregation processes in stochastic hydrology. Water Resources Research 9, pp. 580–585.

    Article  Google Scholar 

  51. Vogel, R.M., Tsai, Y. and Limbrunner, J.F. (1998). The regional persistence and variability of annual steamflow in the United States. Water Resources Research 34, pp. 3445–3459.

    Article  Google Scholar 

  52. Wasimi, S.A. and Kitanidis, P.K. (1983). Real-time forecasting and daily operation of a multi-reservoir system during floods by linear quadratic Gaussian control. Water Resources Research 19, pp. 1511–1522.

    Article  Google Scholar 

  53. Yakowitz, S. (1982). Dynamic programming applications in water resources. Water Resources Research 18, pp. 7673–7696.

    Article  Google Scholar 

  54. Yeh, W.W.G. (1985). Reservoir management and operations models: a state-of-the-art review. Water Resources Research 21, pp. 1797–1818.

    Article  Google Scholar 

  55. Yevjevich, V. (1987). Stochastic models in hydrology. Stochastic Hydrological Hydraulic 1, pp. 17–36.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Département Opérations et Systèmes de Décision, Faculté des Sciences de l’Administration, Université Laval, Québec (QC), G1K 7P4, Canada

    Bernard F. Lamond & Abdeslem Boukhtouta

Authors
  1. Bernard F. Lamond
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abdeslem Boukhtouta
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. State University of New York at Stony Brook, USA

    Eugene A. Feinberg

  2. Technion—Israel Institute of Technology, Israel

    Adam Shwartz

Rights and permissions

Reprints and permissions

Copyright information

© 2002 Springer Science+Business Media New York

About this chapter

Cite this chapter

Lamond, B.F., Boukhtouta, A. (2002). Water Reservoir Applications of Markov Decision Processes. In: Feinberg, E.A., Shwartz, A. (eds) Handbook of Markov Decision Processes. International Series in Operations Research & Management Science, vol 40. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-0805-2_17

Download citation

  • DOI: https://doi.org/10.1007/978-1-4615-0805-2_17

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4613-5248-8

  • Online ISBN: 978-1-4615-0805-2

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation