Controlled approximation of the value function in stochastic dynamic programming for multi-reservoir systems


We present a new approach for adaptive approximation of the value function in stochastic dynamic programming. Under convexity assumptions, our method is based on a simplicial partition of the state space. Bounds on the value function provide guidance as to where refinement should be done, if at all. Thus, the method allows for a trade-off between solution time and accuracy. The proposed scheme is experimented in the particular context of hydroelectric production across multiple reservoirs.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9


  1. Barros MT, Tsai FT, Sl Yang, Lopes JE, Yeh WW (2003) Optimization of large-scale hydropower system operations. J Water Res Plan Manag 129(3):178–188

    Article  Google Scholar 

  2. Castelletti A, de Rigo D, Rizzoli AE, Soncini-Sessa R, Weber E (2007) Neuro-dynamic programming for designing water reservoir network management policies. Control Eng Pract 15(8):1031–1038

    Article  Google Scholar 

  3. Chandramouli V, Raman H (2001) Multireservoir modeling with dynamic programming and neural networks. J Water Res Plan Manag 127(2):89–98

    Article  Google Scholar 

  4. Dickinson PJ (2013) On the exhaustivity of simplicial partitioning. J Glob Optim 58.1:1–15

  5. Foufoula-Georgiou E, Kitanidis PK (1988) Gradient dynamic programming for stochastic optimal control of multidimensional water resources systems. Water Resour Res 24(8):1345–1359

    Article  ADS  Google Scholar 

  6. Gal S (1989) The parameter iteration method in dynamic programming. Manag Sci 35(6):675–684

    MATH  Article  Google Scholar 

  7. Hiriart-Urruty JB, Lemaréchal C (2001) Fundamentals of convex analysis. Springer, Berlin

  8. Kim YO, Eum HI, Lee EG, Ko IH (2007) Optimizing operational policies of a korean multireservoir system using sampling stochastic dynamic programming with ensemble streamflow prediction. J Water Resour Plan Manag 133(1):4–14

    Article  Google Scholar 

  9. Krau S, Émiel G, Merleau J (2014) Une méthode d’échantillonnage adaptatif de l’espace des états pour la programmation dynamique appliquée à la gestion de systèmes hydriques. Seminar presentation, CIRRELT, Montréal, January

  10. Lang P, Zéphyr L, Lamond B (2014) Computing the expected value function for a multi-reservoir system with highly correlated inflows. In: Guan Y, Liao H (eds) Proceedings of the industrial and systems engineering research conference (ISERC 2014), Montreal, Canada, May 31–June 3. Online:

  11. Moore DW (1992) Simplical mesh generation with applications. PhD thesis

  12. Munos R, Moore A (2002) Variable resolution discretization in optimal control. Mach Learn 49(2–3):291–323

    MATH  Article  Google Scholar 

  13. Paulavičius R, Žilinskas J (2013) Simplicial global optimization. Springer, Berlin

  14. Philpott AB, Guan Z (2008) On the convergence of stochastic dual dynamic programming and related methods. Oper Res Lett 36(4):450–455

    MATH  MathSciNet  Article  Google Scholar 

  15. Powell WB (2009) What you should know about approximate dynamic programming. Naval Res Logist 56(3):239–249

    MATH  MathSciNet  Article  Google Scholar 

  16. Rani D, Moreira MM (2010) Simulation-optimization modeling: a survey and potential application in reservoir systems operation. Water Resour Manag 24(6):1107–1138

    Article  Google Scholar 

  17. Shawwash ZK, Siu TK, Russell SD (2000) The BC hydro short term hydro scheduling optimization model. IEEE Trans Power Syst 15(3):1125–1131

    Article  Google Scholar 

  18. Tilmant A, Pinte D, Goor Q (2008) Assessing marginal water values in multipurpose multireservoir systems via stochastic programming. Water Resour Res 44(12):W12,431

  19. Yurtal R, Seckin G, Ardiclioglu G (2005) Hydropower optimization for the lower seyhan system in Turkey using dynamic programming. Water Int 30(4):522–529

    Article  Google Scholar 

  20. Zéphyr L (2015) Optimisation stochastique des systémes multi-réservoirs par l’agrégation de scénarios et la programmation dynamique approximative. PhD thesis

Download references

Author information



Corresponding author

Correspondence to Luckny Zéphyr.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zéphyr, L., Lang, P. & Lamond, B.F. Controlled approximation of the value function in stochastic dynamic programming for multi-reservoir systems. Comput Manag Sci 12, 539–557 (2015).

Download citation


  • Value function approximation
  • Stochastic dynamic programming
  • Simplicial decomposition
  • Regular grids
  • Separable grids
  • Reservoir networks