# Stochastic dynamic programming approach to managing power system uncertainty with distributed storage

- 181 Downloads

## Abstract

Wind integration in power grids is challenging because of the uncertain nature of wind speed. Forecasting errors may have costly consequences. Indeed, power might be purchased at highest prices to meet the load, and in case of surplus, power may be wasted. Energy storage may provide some recourse against the uncertainty of wind generation. Because of their sequential nature, in theory, power scheduling problems may be solved via stochastic dynamic programming. However, this scheme is limited to small networks by the so-called curse of dimensionality. This paper analyzes the management of a network composed of conventional power units and wind turbines through approximate dynamic programming, more precisely stochastic dual dynamic programming. A general power network model with ramping constraints on the conventional generators is considered. The approximate method is tested on several networks of different sizes. The numerical experiments also include comparisons with classical dynamic programming on a small network. The results show that the combination of approximation techniques enables to solve the problem in reasonable time.

## Keywords

Power grid management Energy storage Stochastic dynamic programming Stochastic dual dynamic programming Approximate dynamic programming Generalized linear programming## Notes

### Acknowledgements

This work was supported in part by the National Science Foundation under grant ECCS-1453615. The authors acknowledge constructive comments from two anonymous referees that helped improve the paper.

## References

- Abbey C, Joós G (2009) A stochastic optimization approach to rating of energy storage systems in wind-diesel isolated grids. IEEE Trans Power Syst 24:418–426CrossRefGoogle Scholar
- Anderson RN, Boulanger A, Powell WB, Scott W (2011) Adaptive stochastic control for the smart grid. Proc IEEE 99:1098–1115CrossRefGoogle Scholar
- Bazaraa MS, Sherali HD, Shetty CM (2013) Nonlinear programming: theory and algorithms. Wiley, HobokenGoogle Scholar
- Bellman R (1956) Dynamic programming and lagrange multipliers. In: Proceedings of the National Academy of Sciences of the United States of America 42, 767Google Scholar
- Bellman RE (1957) Some new techniques in the dynamic-programming solution of variational problems. Quart Appl Math 16:295–305CrossRefGoogle Scholar
- Bellman RE, Dreyfus SE (1962) Applied dynamic programming. Princeton university press, PrincetonCrossRefGoogle Scholar
- Bertsekas DP, Tsitsiklis JN (1995) Neuro-dynamic programming: an overview. In: Proceedings of the 34th IEEE conference on decision and control. IEEE, vol. 1, pp. 560–564Google Scholar
- Bradbury K, Pratson L, Patiño-Echeverri D (2014) Economic viability of energy storage systems based on price arbitrage potential in real-time us electricity markets. Appl Energy 114:512–519CrossRefGoogle Scholar
- Castillo A, Gayme DF (2014) Grid-scale energy storage applications in renewable energy integration: a survey. Energy Convers Manag 87:885–894CrossRefGoogle Scholar
- dos Santos Coelho L, Mariani VC (2006) Combining of chaotic differential evolution and quadratic programming for economic dispatch optimization with valve-point effect. IEEE Trans Power Syst 21:989CrossRefGoogle Scholar
- European Commission Impact Assessment (2016). Sustainability of bioenergy, SWD 418. https://ec.europa.eu/energy/sites/ener/files/documents/1_en_impact_assessment_part4_v4_418.pdf
- Foufoula-Georgiou E, Kitanidis PK (1988) Gradient dynamic programming for stochastic optimal control of multidimensional water resources systems. Water Resour Res 24:1345–1359CrossRefGoogle Scholar
- Garcia-Gonzalez J, la Muela D, Ruiz RM, Santos LM, González AM (2008) Stochastic joint optimization of wind generation and pumped-storage units in an electricity market. IEEE Trans Power Syst 23:460–468CrossRefGoogle Scholar
- Goor Q, Kelman R, Tilmant A (2010) Optimal multipurpose-multireservoir operation model with variable productivity of hydropower plants. J Water Resour Plan Manag 137:258–267CrossRefGoogle Scholar
- Grillo S, Marinelli M, Massucco S, Silvestro F (2012) Optimal management strategy of a battery-based storage system to improve renewable energy integration in distribution networks. IEEE Trans Smart Grid 3:950–958CrossRefGoogle Scholar
- Heussen K, Koch S, Ulbig A, Andersson G (2012) Unified system-level modeling of intermittent renewable energy sources and energy storage for power system operation. IEEE Syst J 6:140–151CrossRefGoogle Scholar
- Homem-de Mello T, De Matos VL, Finardi EC (2011) Sampling strategies and stopping criteria for stochastic dual dynamic programming: a case study in long-term hydrothermal scheduling. Energy Syst 2:1–31CrossRefGoogle Scholar
- Howitt R, Msangi S, Reynaud A, Knapp K (2002) Using polynomial approximations to solve stochastic dynamic programming problems: or a ‘betty crocker’ approach to sdp. University of California, DavisGoogle Scholar
- Johnson SA, Stedinger JR, Shoemaker CA, Li Y, Tejada-Guibert JA (1993) Numerical solution of continuous-state dynamic programs using linear and spline interpolation. Oper Res 41:484–500CrossRefGoogle Scholar
- Johri R, Filipi ZS (2011) Self-learning neural controller for hybrid power management using neuro-dynamic programming. Presented at the SAE proceedings, Warrendale, PA, USA, Paper 2011-24-0081Google Scholar
- Keane M, Wolpin K (1994) The solution and estimation of discrete choice dynamic programming models by simulation and interpolation: Monte Carlo evidence. Rev Econ Stat 76:648–672Google Scholar
- Kelley JE Jr (1960) The cutting-plane method for solving convex programs. J Soc Ind Appl Math 8:703–712CrossRefGoogle Scholar
- Khani H, Zadeh MRD (2015) Real-time optimal dispatch and economic viability of cryogenic energy storage exploiting arbitrage opportunities in an electricity market. IEEE Trans Smart Grid 6:391–401CrossRefGoogle Scholar
- Kitanidis P (1987) A first-order approximation to stochastic optimal control of reservoirs. Stoch Hydrol Hydraul 1:169–184CrossRefGoogle Scholar
- Korsak A, Larson R (1970) A dynamic programming successive approximations technique with convergence proofs. Automatica 6:253–260CrossRefGoogle Scholar
- Lee J-H, Labadie JW (2007) Stochastic optimization of multireservoir systems via reinforcement learning. Water Resour Res 43:W11408. https://doi.org/10.1029/2006WR005627CrossRefGoogle Scholar
- Lindenberg S (2009) 20% Wind energy by 2030: increasing wind energy’s contribution to US electricity supply. Diane Publishing, CollingdaleGoogle Scholar
- Löhndorf N, Wozabal D, Minner S (2013) Optimizing trading decisions for hydro storage systems using approximate dual dynamic programming. Oper Res 61:810–823CrossRefGoogle Scholar
- Luh PB, Yu Y, Zhang B, Litvinov E, Zheng T, Zhao F, Zhao J, Wang C (2014) Grid integration of intermittent wind generation: a markovian approach. IEEE Trans Smart Grid 5:732–741CrossRefGoogle Scholar
- Maceira MEP, Duarte V, Penna D, Moraes L, Melo A 2008 Ten years of application of stochastic dual dynamic programming in official and agent studies in Brazil-description of the newave program. In: 16th PSCC, Glasgow, Scotland 14–18Google Scholar
- Mahlia T, Saktisahdan T, Jannifar A, Hasan M, Matseelar H (2014) A review of available methods and development on energy storage; technology update. Renew Sustain Energy Rev 33:532–545CrossRefGoogle Scholar
- Meibom P, Barth R, Hasche B, Brand H, Weber C, O’Malley M (2011) Stochastic optimization model to study the operational impacts of high wind penetrations in ireland. IEEE Trans Power Syst 26:1367–1379CrossRefGoogle Scholar
- Mokrian P, Stephen M 2006 A stochastic programming framework for the valuation of electricity storage. In: 26th USAEE/IAEE North American Conference. 24–27Google Scholar
- Momoh J, Zhang Y et al. (2005) Unit commitment using adaptive dynamic programming. In: Proceedings of the 13th international conference on intelligent systems application to power systemsGoogle Scholar
- Morales-España G, Latorre JM, Ramos A (2013) Tight and compact milp formulation for the thermal unit commitment problem. IEEE Trans Power Syst 28:4897–4908CrossRefGoogle Scholar
- Moura PS, De Almeida AT (2010) The role of demand-side management in the grid integration of wind power. Appl Energy 87:2581–2588CrossRefGoogle Scholar
- Naghibi-Sistani M, Akbarzadeh-Tootoonchi M, Bayaz MJ-D, Rajabi-Mashhadi H (2006) Application of q-learning with temperature variation for bidding strategies in market based power systems. Energy Convers Manage 47:1529–1538CrossRefGoogle Scholar
- NREL (National Renewable Energy Laboratory). http://www.nrel.gov/electricity/transmission/eastern_wind_dataset.html
- NYISO (New York Independent System Operator). Market and operations. http://www.nyiso.com/public/markets_operations/market_data/load_data/index.jsp
- Ostrowski J, Anjos MF, Vannelli A (2012) Tight mixed integer linear programming formulations for the unit commitment problem. IEEE Trans Power Syst 1:39–46CrossRefGoogle Scholar
- Papaefthymiou G, Klockl B (2008) MCMC for wind power simulation. IEEE Trans Energy Convers 23:234–240CrossRefGoogle Scholar
- Papavasiliou A, Oren SS (2013) Multiarea stochastic unit commitment for high wind penetration in a transmission constrained network. Oper Res 61:578–592CrossRefGoogle Scholar
- Park J-B, Lee K-S, Shin J-R, Lee KY (2005) A particle swarm optimization for economic dispatch with nonsmooth cost functions. IEEE Trans Power Syst 20:34–42CrossRefGoogle Scholar
- Pereira M (1989) Optimal stochastic operations scheduling of large hydroelectric systems. Int J Electr Power Energy Syst 11:161–169CrossRefGoogle Scholar
- Pereira M, Pinto L (1985) Stochastic optimization of a multireservoir hydroelectric system: a decomposition approach. Water Resour Res 21:779–792CrossRefGoogle Scholar
- Pereira MV, Pinto LM (1991) Multi-stage stochastic optimization applied to energy planning. Math Progr 52:359–375CrossRefGoogle Scholar
- Pereira-Neto A, Unsihuay C, Saavedra O (2005) Efficient evolutionary strategy optimisation procedure to solve the nonconvex economic dispatch problem with generator constraints. IEE Proc Gen Transm Distrib 152:653–660CrossRefGoogle Scholar
- Philbrick CRJ, Kitanidis PK (2001) Improved dynamic programming methods for optimal control of lumped-parameter stochastic systems. Oper Res 49:398–412CrossRefGoogle Scholar
- Philpott AB, de Matos VL (2012) Dynamic sampling algorithms for multi-stage stochastic programs with risk aversion. Eur J Oper Res 218:470–483CrossRefGoogle Scholar
- Philpott AB, Guan Z (2008) On the convergence of stochastic dual dynamic programming and related methods. Oper Res Lett 36:450–455CrossRefGoogle Scholar
- Pinson P et al (2013) Wind energy: forecasting challenges for its operational management. Stat Sci 28:564–585CrossRefGoogle Scholar
- Qiu Q, Pedram M (1999) Dynamic power management based on continuous-time markov decision processes. In: Proceedings of the 36th annual ACM/IEEE design automation conference. ACM, 555–561Google Scholar
- Renewable energy policy network for the 21st century. 2017. Renewables 2017, Global status report. technical reportGoogle Scholar
- Sayah S, Zehar K (2008) Modified differential evolution algorithm for optimal power flow with non-smooth cost functions. Energy Convers Manage 49:3036–3042CrossRefGoogle Scholar
- Schoenung S (2011) Energy storage systems cost updateGoogle Scholar
- Scott W, Powell WB (2012) Approximate dynamic programming for energy storage with new results on instrumental variables and projected bellman errors. Submitted to Operations Research (Under Review)Google Scholar
- Shapiro A (2011) Analysis of stochastic dual dynamic programming method. Eur J Oper Res 209:63–72CrossRefGoogle Scholar
- Shapiro A, Tekaya W, da Costa JP, Soares MP (2013) Risk neutral and risk averse stochastic dual dynamic programming method. Eur J Oper Res 224:375–391CrossRefGoogle Scholar
- Shapiro JF (1979) Mathematical programming: structures and algorithms. Wiley, New YorkGoogle Scholar
- Song H, Liu C-C, Lawarrée J, Dahlgren RW (2000) Optimal electricity supply bidding by markov decision process. IEEE Trans Power Syst 15:618–624CrossRefGoogle Scholar
- Suberu MY, Mustafa MW, Bashir N (2014) Energy storage systems for renewable energy power sector integration and mitigation of intermittency. Renew Sustain Energy Rev 35:499–514CrossRefGoogle Scholar
- Succar S, Denkenberger DC, Williams RH (2012) Optimization of specific rating for wind turbine arrays coupled to compressed air energy storage. Appl Energy 96:222–234CrossRefGoogle Scholar
- Tan X, Li Q, Wang H (2013) Advances and trends of energy storage technology in microgrid. Int J Electr Power Energy Syst 44:179–191CrossRefGoogle Scholar
- Tan Y, Liu W, Qiu Q (2009) Adaptive power management using reinforcement learning. In: Proceedings of the 2009 international conference on computer-aided design. ACM, 461–467Google Scholar
- Thatte AA, Xie L, Viassolo DE, Singh S (2013) Risk measure based robust bidding strategy for arbitrage using a wind farm and energy storage. IEEE Trans Smart Grid 4:2191–2199CrossRefGoogle Scholar
- Topaloglu H, Powell WB (2006) Dynamic-programming approximations for stochastic time-staged integer multicommodity-flow problems. INFORMS J Comput 18:31–42CrossRefGoogle Scholar
- Turgeon A (1980) Optimal operation of multireservoir power systems with stochastic inflows. Water Resour Res 16:275–283CrossRefGoogle Scholar
- Wee J-H (2013) A review on carbon dioxide capture and storage technology using coal fly ash. Appl Energy 106:143–151CrossRefGoogle Scholar
- Yang J-S, Chen N (1989) Short term hydrothermal coordination using multi-pass dynamic programming. IEEE Trans Power Syst 4:1050–1056CrossRefGoogle Scholar
- Yu Y, Luh PB, Litvinov E, Zheng T, Zhao J, Zhao F (2015) Grid integration of distributed wind generation: hybrid markovian and interval unit commitment. IEEE Trans Smart Grid 6:3061–3072CrossRefGoogle Scholar
- Zéphyr L, Lang P, Lamond BF (2015) Controlled approximation of the value function in stochastic dynamic programming for multi-reservoir systems. CMS 12:539–557CrossRefGoogle Scholar
- Zéphyr L, Lang P, Lamond BF, Côté P (2017) Approximate stochastic dynamic programming for hydroelectric production planning. Eur J Oper Res 262:586–601CrossRefGoogle Scholar
- Zhou Y, Scheller-Wolf AA, Secomandi N, Smith S (2014) Managing wind-based electricity generation in the presence of storage and transmission capacity. SSRN 1962414Google Scholar
- Zurn H, Quintana V (1975) Generator maintenance scheduling via successive approximations dynamic programming. IEEE Trans Power Appar Syst 94:665–671CrossRefGoogle Scholar