Computational Management Science

, Volume 15, Issue 1, pp 87–110 | Cite as

Stochastic dynamic programming approach to managing power system uncertainty with distributed storage

  • Luckny Zéphyr
  • C. Lindsay Anderson
Original Paper


Wind integration in power grids is challenging because of the uncertain nature of wind speed. Forecasting errors may have costly consequences. Indeed, power might be purchased at highest prices to meet the load, and in case of surplus, power may be wasted. Energy storage may provide some recourse against the uncertainty of wind generation. Because of their sequential nature, in theory, power scheduling problems may be solved via stochastic dynamic programming. However, this scheme is limited to small networks by the so-called curse of dimensionality. This paper analyzes the management of a network composed of conventional power units and wind turbines through approximate dynamic programming, more precisely stochastic dual dynamic programming. A general power network model with ramping constraints on the conventional generators is considered. The approximate method is tested on several networks of different sizes. The numerical experiments also include comparisons with classical dynamic programming on a small network. The results show that the combination of approximation techniques enables to solve the problem in reasonable time.


Power grid management Energy storage Stochastic dynamic programming Stochastic dual dynamic programming Approximate dynamic programming Generalized linear programming 



This work was supported in part by the National Science Foundation under grant ECCS-1453615. The authors acknowledge constructive comments from two anonymous referees that helped improve the paper.


  1. Abbey C, Joós G (2009) A stochastic optimization approach to rating of energy storage systems in wind-diesel isolated grids. IEEE Trans Power Syst 24:418–426CrossRefGoogle Scholar
  2. Anderson RN, Boulanger A, Powell WB, Scott W (2011) Adaptive stochastic control for the smart grid. Proc IEEE 99:1098–1115CrossRefGoogle Scholar
  3. Bazaraa MS, Sherali HD, Shetty CM (2013) Nonlinear programming: theory and algorithms. Wiley, HobokenGoogle Scholar
  4. Bellman R (1956) Dynamic programming and lagrange multipliers. In: Proceedings of the National Academy of Sciences of the United States of America 42, 767Google Scholar
  5. Bellman RE (1957) Some new techniques in the dynamic-programming solution of variational problems. Quart Appl Math 16:295–305CrossRefGoogle Scholar
  6. Bellman RE, Dreyfus SE (1962) Applied dynamic programming. Princeton university press, PrincetonCrossRefGoogle Scholar
  7. Bertsekas DP, Tsitsiklis JN (1995) Neuro-dynamic programming: an overview. In: Proceedings of the 34th IEEE conference on decision and control. IEEE, vol. 1, pp. 560–564Google Scholar
  8. Bradbury K, Pratson L, Patiño-Echeverri D (2014) Economic viability of energy storage systems based on price arbitrage potential in real-time us electricity markets. Appl Energy 114:512–519CrossRefGoogle Scholar
  9. Castillo A, Gayme DF (2014) Grid-scale energy storage applications in renewable energy integration: a survey. Energy Convers Manag 87:885–894CrossRefGoogle Scholar
  10. dos Santos Coelho L, Mariani VC (2006) Combining of chaotic differential evolution and quadratic programming for economic dispatch optimization with valve-point effect. IEEE Trans Power Syst 21:989CrossRefGoogle Scholar
  11. European Commission Impact Assessment (2016). Sustainability of bioenergy, SWD 418.
  12. Foufoula-Georgiou E, Kitanidis PK (1988) Gradient dynamic programming for stochastic optimal control of multidimensional water resources systems. Water Resour Res 24:1345–1359CrossRefGoogle Scholar
  13. Garcia-Gonzalez J, la Muela D, Ruiz RM, Santos LM, González AM (2008) Stochastic joint optimization of wind generation and pumped-storage units in an electricity market. IEEE Trans Power Syst 23:460–468CrossRefGoogle Scholar
  14. Goor Q, Kelman R, Tilmant A (2010) Optimal multipurpose-multireservoir operation model with variable productivity of hydropower plants. J Water Resour Plan Manag 137:258–267CrossRefGoogle Scholar
  15. Grillo S, Marinelli M, Massucco S, Silvestro F (2012) Optimal management strategy of a battery-based storage system to improve renewable energy integration in distribution networks. IEEE Trans Smart Grid 3:950–958CrossRefGoogle Scholar
  16. Heussen K, Koch S, Ulbig A, Andersson G (2012) Unified system-level modeling of intermittent renewable energy sources and energy storage for power system operation. IEEE Syst J 6:140–151CrossRefGoogle Scholar
  17. Homem-de Mello T, De Matos VL, Finardi EC (2011) Sampling strategies and stopping criteria for stochastic dual dynamic programming: a case study in long-term hydrothermal scheduling. Energy Syst 2:1–31CrossRefGoogle Scholar
  18. Howitt R, Msangi S, Reynaud A, Knapp K (2002) Using polynomial approximations to solve stochastic dynamic programming problems: or a ‘betty crocker’ approach to sdp. University of California, DavisGoogle Scholar
  19. Johnson SA, Stedinger JR, Shoemaker CA, Li Y, Tejada-Guibert JA (1993) Numerical solution of continuous-state dynamic programs using linear and spline interpolation. Oper Res 41:484–500CrossRefGoogle Scholar
  20. Johri R, Filipi ZS (2011) Self-learning neural controller for hybrid power management using neuro-dynamic programming. Presented at the SAE proceedings, Warrendale, PA, USA, Paper 2011-24-0081Google Scholar
  21. Keane M, Wolpin K (1994) The solution and estimation of discrete choice dynamic programming models by simulation and interpolation: Monte Carlo evidence. Rev Econ Stat 76:648–672Google Scholar
  22. Kelley JE Jr (1960) The cutting-plane method for solving convex programs. J Soc Ind Appl Math 8:703–712CrossRefGoogle Scholar
  23. Khani H, Zadeh MRD (2015) Real-time optimal dispatch and economic viability of cryogenic energy storage exploiting arbitrage opportunities in an electricity market. IEEE Trans Smart Grid 6:391–401CrossRefGoogle Scholar
  24. Kitanidis P (1987) A first-order approximation to stochastic optimal control of reservoirs. Stoch Hydrol Hydraul 1:169–184CrossRefGoogle Scholar
  25. Korsak A, Larson R (1970) A dynamic programming successive approximations technique with convergence proofs. Automatica 6:253–260CrossRefGoogle Scholar
  26. Lee J-H, Labadie JW (2007) Stochastic optimization of multireservoir systems via reinforcement learning. Water Resour Res 43:W11408. Scholar
  27. Lindenberg S (2009) 20% Wind energy by 2030: increasing wind energy’s contribution to US electricity supply. Diane Publishing, CollingdaleGoogle Scholar
  28. Löhndorf N, Wozabal D, Minner S (2013) Optimizing trading decisions for hydro storage systems using approximate dual dynamic programming. Oper Res 61:810–823CrossRefGoogle Scholar
  29. Luh PB, Yu Y, Zhang B, Litvinov E, Zheng T, Zhao F, Zhao J, Wang C (2014) Grid integration of intermittent wind generation: a markovian approach. IEEE Trans Smart Grid 5:732–741CrossRefGoogle Scholar
  30. Maceira MEP, Duarte V, Penna D, Moraes L, Melo A 2008 Ten years of application of stochastic dual dynamic programming in official and agent studies in Brazil-description of the newave program. In: 16th PSCC, Glasgow, Scotland 14–18Google Scholar
  31. Mahlia T, Saktisahdan T, Jannifar A, Hasan M, Matseelar H (2014) A review of available methods and development on energy storage; technology update. Renew Sustain Energy Rev 33:532–545CrossRefGoogle Scholar
  32. Meibom P, Barth R, Hasche B, Brand H, Weber C, O’Malley M (2011) Stochastic optimization model to study the operational impacts of high wind penetrations in ireland. IEEE Trans Power Syst 26:1367–1379CrossRefGoogle Scholar
  33. Mokrian P, Stephen M 2006 A stochastic programming framework for the valuation of electricity storage. In: 26th USAEE/IAEE North American Conference. 24–27Google Scholar
  34. Momoh J, Zhang Y et al. (2005) Unit commitment using adaptive dynamic programming. In: Proceedings of the 13th international conference on intelligent systems application to power systemsGoogle Scholar
  35. Morales-España G, Latorre JM, Ramos A (2013) Tight and compact milp formulation for the thermal unit commitment problem. IEEE Trans Power Syst 28:4897–4908CrossRefGoogle Scholar
  36. Moura PS, De Almeida AT (2010) The role of demand-side management in the grid integration of wind power. Appl Energy 87:2581–2588CrossRefGoogle Scholar
  37. Naghibi-Sistani M, Akbarzadeh-Tootoonchi M, Bayaz MJ-D, Rajabi-Mashhadi H (2006) Application of q-learning with temperature variation for bidding strategies in market based power systems. Energy Convers Manage 47:1529–1538CrossRefGoogle Scholar
  38. NYISO (New York Independent System Operator). Market and operations.
  39. Ostrowski J, Anjos MF, Vannelli A (2012) Tight mixed integer linear programming formulations for the unit commitment problem. IEEE Trans Power Syst 1:39–46CrossRefGoogle Scholar
  40. Papaefthymiou G, Klockl B (2008) MCMC for wind power simulation. IEEE Trans Energy Convers 23:234–240CrossRefGoogle Scholar
  41. Papavasiliou A, Oren SS (2013) Multiarea stochastic unit commitment for high wind penetration in a transmission constrained network. Oper Res 61:578–592CrossRefGoogle Scholar
  42. Park J-B, Lee K-S, Shin J-R, Lee KY (2005) A particle swarm optimization for economic dispatch with nonsmooth cost functions. IEEE Trans Power Syst 20:34–42CrossRefGoogle Scholar
  43. Pereira M (1989) Optimal stochastic operations scheduling of large hydroelectric systems. Int J Electr Power Energy Syst 11:161–169CrossRefGoogle Scholar
  44. Pereira M, Pinto L (1985) Stochastic optimization of a multireservoir hydroelectric system: a decomposition approach. Water Resour Res 21:779–792CrossRefGoogle Scholar
  45. Pereira MV, Pinto LM (1991) Multi-stage stochastic optimization applied to energy planning. Math Progr 52:359–375CrossRefGoogle Scholar
  46. Pereira-Neto A, Unsihuay C, Saavedra O (2005) Efficient evolutionary strategy optimisation procedure to solve the nonconvex economic dispatch problem with generator constraints. IEE Proc Gen Transm Distrib 152:653–660CrossRefGoogle Scholar
  47. Philbrick CRJ, Kitanidis PK (2001) Improved dynamic programming methods for optimal control of lumped-parameter stochastic systems. Oper Res 49:398–412CrossRefGoogle Scholar
  48. Philpott AB, de Matos VL (2012) Dynamic sampling algorithms for multi-stage stochastic programs with risk aversion. Eur J Oper Res 218:470–483CrossRefGoogle Scholar
  49. Philpott AB, Guan Z (2008) On the convergence of stochastic dual dynamic programming and related methods. Oper Res Lett 36:450–455CrossRefGoogle Scholar
  50. Pinson P et al (2013) Wind energy: forecasting challenges for its operational management. Stat Sci 28:564–585CrossRefGoogle Scholar
  51. Qiu Q, Pedram M (1999) Dynamic power management based on continuous-time markov decision processes. In: Proceedings of the 36th annual ACM/IEEE design automation conference. ACM, 555–561Google Scholar
  52. Renewable energy policy network for the 21st century. 2017. Renewables 2017, Global status report. technical reportGoogle Scholar
  53. Sayah S, Zehar K (2008) Modified differential evolution algorithm for optimal power flow with non-smooth cost functions. Energy Convers Manage 49:3036–3042CrossRefGoogle Scholar
  54. Schoenung S (2011) Energy storage systems cost updateGoogle Scholar
  55. Scott W, Powell WB (2012) Approximate dynamic programming for energy storage with new results on instrumental variables and projected bellman errors. Submitted to Operations Research (Under Review)Google Scholar
  56. Shapiro A (2011) Analysis of stochastic dual dynamic programming method. Eur J Oper Res 209:63–72CrossRefGoogle Scholar
  57. Shapiro A, Tekaya W, da Costa JP, Soares MP (2013) Risk neutral and risk averse stochastic dual dynamic programming method. Eur J Oper Res 224:375–391CrossRefGoogle Scholar
  58. Shapiro JF (1979) Mathematical programming: structures and algorithms. Wiley, New YorkGoogle Scholar
  59. Song H, Liu C-C, Lawarrée J, Dahlgren RW (2000) Optimal electricity supply bidding by markov decision process. IEEE Trans Power Syst 15:618–624CrossRefGoogle Scholar
  60. Suberu MY, Mustafa MW, Bashir N (2014) Energy storage systems for renewable energy power sector integration and mitigation of intermittency. Renew Sustain Energy Rev 35:499–514CrossRefGoogle Scholar
  61. Succar S, Denkenberger DC, Williams RH (2012) Optimization of specific rating for wind turbine arrays coupled to compressed air energy storage. Appl Energy 96:222–234CrossRefGoogle Scholar
  62. Tan X, Li Q, Wang H (2013) Advances and trends of energy storage technology in microgrid. Int J Electr Power Energy Syst 44:179–191CrossRefGoogle Scholar
  63. Tan Y, Liu W, Qiu Q (2009) Adaptive power management using reinforcement learning. In: Proceedings of the 2009 international conference on computer-aided design. ACM, 461–467Google Scholar
  64. Thatte AA, Xie L, Viassolo DE, Singh S (2013) Risk measure based robust bidding strategy for arbitrage using a wind farm and energy storage. IEEE Trans Smart Grid 4:2191–2199CrossRefGoogle Scholar
  65. Topaloglu H, Powell WB (2006) Dynamic-programming approximations for stochastic time-staged integer multicommodity-flow problems. INFORMS J Comput 18:31–42CrossRefGoogle Scholar
  66. Turgeon A (1980) Optimal operation of multireservoir power systems with stochastic inflows. Water Resour Res 16:275–283CrossRefGoogle Scholar
  67. Wee J-H (2013) A review on carbon dioxide capture and storage technology using coal fly ash. Appl Energy 106:143–151CrossRefGoogle Scholar
  68. Yang J-S, Chen N (1989) Short term hydrothermal coordination using multi-pass dynamic programming. IEEE Trans Power Syst 4:1050–1056CrossRefGoogle Scholar
  69. Yu Y, Luh PB, Litvinov E, Zheng T, Zhao J, Zhao F (2015) Grid integration of distributed wind generation: hybrid markovian and interval unit commitment. IEEE Trans Smart Grid 6:3061–3072CrossRefGoogle Scholar
  70. Zéphyr L, Lang P, Lamond BF (2015) Controlled approximation of the value function in stochastic dynamic programming for multi-reservoir systems. CMS 12:539–557CrossRefGoogle Scholar
  71. Zéphyr L, Lang P, Lamond BF, Côté P (2017) Approximate stochastic dynamic programming for hydroelectric production planning. Eur J Oper Res 262:586–601CrossRefGoogle Scholar
  72. Zhou Y, Scheller-Wolf AA, Secomandi N, Smith S (2014) Managing wind-based electricity generation in the presence of storage and transmission capacity. SSRN 1962414Google Scholar
  73. Zurn H, Quintana V (1975) Generator maintenance scheduling via successive approximations dynamic programming. IEEE Trans Power Appar Syst 94:665–671CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Department of Biological and Environmental EngineeringCornell UniversityIthacaUSA

Personalised recommendations