Abstract
Wind integration in power grids is challenging because of the uncertain nature of wind speed. Forecasting errors may have costly consequences. Indeed, power might be purchased at highest prices to meet the load, and in case of surplus, power may be wasted. Energy storage may provide some recourse against the uncertainty of wind generation. Because of their sequential nature, in theory, power scheduling problems may be solved via stochastic dynamic programming. However, this scheme is limited to small networks by the so-called curse of dimensionality. This paper analyzes the management of a network composed of conventional power units and wind turbines through approximate dynamic programming, more precisely stochastic dual dynamic programming. A general power network model with ramping constraints on the conventional generators is considered. The approximate method is tested on several networks of different sizes. The numerical experiments also include comparisons with classical dynamic programming on a small network. The results show that the combination of approximation techniques enables to solve the problem in reasonable time.
Similar content being viewed by others
References
Abbey C, Joós G (2009) A stochastic optimization approach to rating of energy storage systems in wind-diesel isolated grids. IEEE Trans Power Syst 24:418–426
Anderson RN, Boulanger A, Powell WB, Scott W (2011) Adaptive stochastic control for the smart grid. Proc IEEE 99:1098–1115
Bazaraa MS, Sherali HD, Shetty CM (2013) Nonlinear programming: theory and algorithms. Wiley, Hoboken
Bellman R (1956) Dynamic programming and lagrange multipliers. In: Proceedings of the National Academy of Sciences of the United States of America 42, 767
Bellman RE (1957) Some new techniques in the dynamic-programming solution of variational problems. Quart Appl Math 16:295–305
Bellman RE, Dreyfus SE (1962) Applied dynamic programming. Princeton university press, Princeton
Bertsekas DP, Tsitsiklis JN (1995) Neuro-dynamic programming: an overview. In: Proceedings of the 34th IEEE conference on decision and control. IEEE, vol. 1, pp. 560–564
Bradbury K, Pratson L, Patiño-Echeverri D (2014) Economic viability of energy storage systems based on price arbitrage potential in real-time us electricity markets. Appl Energy 114:512–519
Castillo A, Gayme DF (2014) Grid-scale energy storage applications in renewable energy integration: a survey. Energy Convers Manag 87:885–894
dos Santos Coelho L, Mariani VC (2006) Combining of chaotic differential evolution and quadratic programming for economic dispatch optimization with valve-point effect. IEEE Trans Power Syst 21:989
European Commission Impact Assessment (2016). Sustainability of bioenergy, SWD 418. https://ec.europa.eu/energy/sites/ener/files/documents/1_en_impact_assessment_part4_v4_418.pdf
Foufoula-Georgiou E, Kitanidis PK (1988) Gradient dynamic programming for stochastic optimal control of multidimensional water resources systems. Water Resour Res 24:1345–1359
Garcia-Gonzalez J, la Muela D, Ruiz RM, Santos LM, González AM (2008) Stochastic joint optimization of wind generation and pumped-storage units in an electricity market. IEEE Trans Power Syst 23:460–468
Goor Q, Kelman R, Tilmant A (2010) Optimal multipurpose-multireservoir operation model with variable productivity of hydropower plants. J Water Resour Plan Manag 137:258–267
Grillo S, Marinelli M, Massucco S, Silvestro F (2012) Optimal management strategy of a battery-based storage system to improve renewable energy integration in distribution networks. IEEE Trans Smart Grid 3:950–958
Heussen K, Koch S, Ulbig A, Andersson G (2012) Unified system-level modeling of intermittent renewable energy sources and energy storage for power system operation. IEEE Syst J 6:140–151
Homem-de Mello T, De Matos VL, Finardi EC (2011) Sampling strategies and stopping criteria for stochastic dual dynamic programming: a case study in long-term hydrothermal scheduling. Energy Syst 2:1–31
Howitt R, Msangi S, Reynaud A, Knapp K (2002) Using polynomial approximations to solve stochastic dynamic programming problems: or a ‘betty crocker’ approach to sdp. University of California, Davis
Johnson SA, Stedinger JR, Shoemaker CA, Li Y, Tejada-Guibert JA (1993) Numerical solution of continuous-state dynamic programs using linear and spline interpolation. Oper Res 41:484–500
Johri R, Filipi ZS (2011) Self-learning neural controller for hybrid power management using neuro-dynamic programming. Presented at the SAE proceedings, Warrendale, PA, USA, Paper 2011-24-0081
Keane M, Wolpin K (1994) The solution and estimation of discrete choice dynamic programming models by simulation and interpolation: Monte Carlo evidence. Rev Econ Stat 76:648–672
Kelley JE Jr (1960) The cutting-plane method for solving convex programs. J Soc Ind Appl Math 8:703–712
Khani H, Zadeh MRD (2015) Real-time optimal dispatch and economic viability of cryogenic energy storage exploiting arbitrage opportunities in an electricity market. IEEE Trans Smart Grid 6:391–401
Kitanidis P (1987) A first-order approximation to stochastic optimal control of reservoirs. Stoch Hydrol Hydraul 1:169–184
Korsak A, Larson R (1970) A dynamic programming successive approximations technique with convergence proofs. Automatica 6:253–260
Lee J-H, Labadie JW (2007) Stochastic optimization of multireservoir systems via reinforcement learning. Water Resour Res 43:W11408. https://doi.org/10.1029/2006WR005627
Lindenberg S (2009) 20% Wind energy by 2030: increasing wind energy’s contribution to US electricity supply. Diane Publishing, Collingdale
Löhndorf N, Wozabal D, Minner S (2013) Optimizing trading decisions for hydro storage systems using approximate dual dynamic programming. Oper Res 61:810–823
Luh PB, Yu Y, Zhang B, Litvinov E, Zheng T, Zhao F, Zhao J, Wang C (2014) Grid integration of intermittent wind generation: a markovian approach. IEEE Trans Smart Grid 5:732–741
Maceira MEP, Duarte V, Penna D, Moraes L, Melo A 2008 Ten years of application of stochastic dual dynamic programming in official and agent studies in Brazil-description of the newave program. In: 16th PSCC, Glasgow, Scotland 14–18
Mahlia T, Saktisahdan T, Jannifar A, Hasan M, Matseelar H (2014) A review of available methods and development on energy storage; technology update. Renew Sustain Energy Rev 33:532–545
Meibom P, Barth R, Hasche B, Brand H, Weber C, O’Malley M (2011) Stochastic optimization model to study the operational impacts of high wind penetrations in ireland. IEEE Trans Power Syst 26:1367–1379
Mokrian P, Stephen M 2006 A stochastic programming framework for the valuation of electricity storage. In: 26th USAEE/IAEE North American Conference. 24–27
Momoh J, Zhang Y et al. (2005) Unit commitment using adaptive dynamic programming. In: Proceedings of the 13th international conference on intelligent systems application to power systems
Morales-España G, Latorre JM, Ramos A (2013) Tight and compact milp formulation for the thermal unit commitment problem. IEEE Trans Power Syst 28:4897–4908
Moura PS, De Almeida AT (2010) The role of demand-side management in the grid integration of wind power. Appl Energy 87:2581–2588
Naghibi-Sistani M, Akbarzadeh-Tootoonchi M, Bayaz MJ-D, Rajabi-Mashhadi H (2006) Application of q-learning with temperature variation for bidding strategies in market based power systems. Energy Convers Manage 47:1529–1538
NREL (National Renewable Energy Laboratory). http://www.nrel.gov/electricity/transmission/eastern_wind_dataset.html
NYISO (New York Independent System Operator). Market and operations. http://www.nyiso.com/public/markets_operations/market_data/load_data/index.jsp
Ostrowski J, Anjos MF, Vannelli A (2012) Tight mixed integer linear programming formulations for the unit commitment problem. IEEE Trans Power Syst 1:39–46
Papaefthymiou G, Klockl B (2008) MCMC for wind power simulation. IEEE Trans Energy Convers 23:234–240
Papavasiliou A, Oren SS (2013) Multiarea stochastic unit commitment for high wind penetration in a transmission constrained network. Oper Res 61:578–592
Park J-B, Lee K-S, Shin J-R, Lee KY (2005) A particle swarm optimization for economic dispatch with nonsmooth cost functions. IEEE Trans Power Syst 20:34–42
Pereira M (1989) Optimal stochastic operations scheduling of large hydroelectric systems. Int J Electr Power Energy Syst 11:161–169
Pereira M, Pinto L (1985) Stochastic optimization of a multireservoir hydroelectric system: a decomposition approach. Water Resour Res 21:779–792
Pereira MV, Pinto LM (1991) Multi-stage stochastic optimization applied to energy planning. Math Progr 52:359–375
Pereira-Neto A, Unsihuay C, Saavedra O (2005) Efficient evolutionary strategy optimisation procedure to solve the nonconvex economic dispatch problem with generator constraints. IEE Proc Gen Transm Distrib 152:653–660
Philbrick CRJ, Kitanidis PK (2001) Improved dynamic programming methods for optimal control of lumped-parameter stochastic systems. Oper Res 49:398–412
Philpott AB, de Matos VL (2012) Dynamic sampling algorithms for multi-stage stochastic programs with risk aversion. Eur J Oper Res 218:470–483
Philpott AB, Guan Z (2008) On the convergence of stochastic dual dynamic programming and related methods. Oper Res Lett 36:450–455
Pinson P et al (2013) Wind energy: forecasting challenges for its operational management. Stat Sci 28:564–585
Qiu Q, Pedram M (1999) Dynamic power management based on continuous-time markov decision processes. In: Proceedings of the 36th annual ACM/IEEE design automation conference. ACM, 555–561
Renewable energy policy network for the 21st century. 2017. Renewables 2017, Global status report. technical report
Sayah S, Zehar K (2008) Modified differential evolution algorithm for optimal power flow with non-smooth cost functions. Energy Convers Manage 49:3036–3042
Schoenung S (2011) Energy storage systems cost update
Scott W, Powell WB (2012) Approximate dynamic programming for energy storage with new results on instrumental variables and projected bellman errors. Submitted to Operations Research (Under Review)
Shapiro A (2011) Analysis of stochastic dual dynamic programming method. Eur J Oper Res 209:63–72
Shapiro A, Tekaya W, da Costa JP, Soares MP (2013) Risk neutral and risk averse stochastic dual dynamic programming method. Eur J Oper Res 224:375–391
Shapiro JF (1979) Mathematical programming: structures and algorithms. Wiley, New York
Song H, Liu C-C, Lawarrée J, Dahlgren RW (2000) Optimal electricity supply bidding by markov decision process. IEEE Trans Power Syst 15:618–624
Suberu MY, Mustafa MW, Bashir N (2014) Energy storage systems for renewable energy power sector integration and mitigation of intermittency. Renew Sustain Energy Rev 35:499–514
Succar S, Denkenberger DC, Williams RH (2012) Optimization of specific rating for wind turbine arrays coupled to compressed air energy storage. Appl Energy 96:222–234
Tan X, Li Q, Wang H (2013) Advances and trends of energy storage technology in microgrid. Int J Electr Power Energy Syst 44:179–191
Tan Y, Liu W, Qiu Q (2009) Adaptive power management using reinforcement learning. In: Proceedings of the 2009 international conference on computer-aided design. ACM, 461–467
Thatte AA, Xie L, Viassolo DE, Singh S (2013) Risk measure based robust bidding strategy for arbitrage using a wind farm and energy storage. IEEE Trans Smart Grid 4:2191–2199
Topaloglu H, Powell WB (2006) Dynamic-programming approximations for stochastic time-staged integer multicommodity-flow problems. INFORMS J Comput 18:31–42
Turgeon A (1980) Optimal operation of multireservoir power systems with stochastic inflows. Water Resour Res 16:275–283
Wee J-H (2013) A review on carbon dioxide capture and storage technology using coal fly ash. Appl Energy 106:143–151
Yang J-S, Chen N (1989) Short term hydrothermal coordination using multi-pass dynamic programming. IEEE Trans Power Syst 4:1050–1056
Yu Y, Luh PB, Litvinov E, Zheng T, Zhao J, Zhao F (2015) Grid integration of distributed wind generation: hybrid markovian and interval unit commitment. IEEE Trans Smart Grid 6:3061–3072
Zéphyr L, Lang P, Lamond BF (2015) Controlled approximation of the value function in stochastic dynamic programming for multi-reservoir systems. CMS 12:539–557
Zéphyr L, Lang P, Lamond BF, Côté P (2017) Approximate stochastic dynamic programming for hydroelectric production planning. Eur J Oper Res 262:586–601
Zhou Y, Scheller-Wolf AA, Secomandi N, Smith S (2014) Managing wind-based electricity generation in the presence of storage and transmission capacity. SSRN 1962414
Zurn H, Quintana V (1975) Generator maintenance scheduling via successive approximations dynamic programming. IEEE Trans Power Appar Syst 94:665–671
Acknowledgements
This work was supported in part by the National Science Foundation under grant ECCS-1453615. The authors acknowledge constructive comments from two anonymous referees that helped improve the paper.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
1.1 Nomenclature
1.1.1 Sets
- N :
-
Buses
- L :
-
Transmission lines
- M :
-
Wind farms
- G :
-
Conventional generators
- \(G_n\) :
-
Conventional generators at bus n
- S :
-
Storage devices
- \(O_n\) :
-
Transmission lines that leave node n
- \(I_n\) :
-
Transmission lines that enter node n
- \(\mathbf{R}\) :
-
Real numbers
- \(\mathbf{R}^+\) :
-
Non negative numbers
1.1.2 Parameters
- T :
-
Length of the planning horizon
- t :
-
Time period
- \({\hat{D}}_{nt}\) :
-
Forecast of the load, in period t at bus n
- \({\underline{p}}_g\) :
-
(resp. \({\bar{p}}_g\)) Lower (resp. upper) bound on generator g output
- \({\underline{\lambda }}_g\) :
-
(resp. \({\bar{\lambda }}_g\)) Ramp down (resp. ramp up) limit on generator g
- \({\tilde{W}}_t\) :
-
Random vector of outputs from the wind turbines in period t
- \(w_{mt}\) :
-
A particular realization of the stochastic process \(\{{\tilde{W}}_{mt}\}\)
- \({\bar{e}}_l\) :
-
Bound on power trough line l
- \(B_l\) :
-
Susceptance of line l
- \({\underline{s}}_n\) :
-
(resp. \({\bar{s}}_n\)) Lower (resp. upper) bound on the level of the storage at bus n
- \(c_n\) :
-
(resp. \(d_n\)) Efficiency coefficient of charging (resp. discharging) of the storage device at bus n
- \(\delta _{n}=\) :
-
\(\left\{ \begin{array}{ll} 1 &{} \hbox {if there is a storage facility at node n},\\ 0 &{} \hbox {otherwise}. \end{array} \right. \)
1.1.3 Decision variables
- \(p_{gt}\) :
-
Power generation from generator g in period t
- \(e_{lt}\) :
-
Power flowing through transmission line l in period t
- \(\theta _{nt}\) :
-
Phase angle of bus n in period t
- \(s_{nt}\) :
-
Level of stored energy at bus n in the beginning of period t
- \(\Delta ^+_{nt}\) :
-
(resp. \(\Delta ^-_{nt}\)) Positive (resp. negative) variation in the level of charge from the beginning through the end of period t
- \(\kappa ^+_{nt}\) :
-
(resp. \(\kappa ^-_{nt}\)) Power excess absorbed (resp. delivered) by the storage unit located at bus n
- \(\Delta _{nt}\) :
-
Power absorbed or delivered by the storage unit at bus n
1.1.4 Functions
- \(CP_{gt} (p_{gt})\) :
-
Cost function of generator g in period t
- \(CS_{nt}(\Delta _{nt})\) :
-
Cost associated with varying the stored energy at bus n
1.1.5 Operators
- \({\mathbb {E}}\) :
-
Mathematical expectation
- |X|:
-
Cardinality of the set X
Rights and permissions
About this article
Cite this article
Zéphyr, L., Anderson, C.L. Stochastic dynamic programming approach to managing power system uncertainty with distributed storage. Comput Manag Sci 15, 87–110 (2018). https://doi.org/10.1007/s10287-017-0297-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10287-017-0297-2