Skip to main content

Advertisement

SpringerLink
Log in

Rolling horizon wind-thermal unit commitment optimization based on deep reinforcement learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The growing penetration of renewable energy has brought significant challenges for modern power system operation. Academic research and industrial practice show that adjusting unit commitment (UC) scheduling periodically according to new forecasts of renewable power provides a promising way to improve system stability and economy; however, this greatly increases the computational burden for solution methods. In this paper, a deep reinforcement learning (DRL) method is proposed to obtain timely and reliable solutions for rolling-horizon UC (RHUC). First, based on historical data and day-ahead point forecasting, a data-driven method is designed to construct typical wind power scenarios that are regarded as components of the state space of DRL. Second, a rolling mechanism is proposed to dynamically update the state space based on real-time wind power data. Third, unlike existing reinforcement learning-based UC solution methods that segment the continuous outputs of generators as discrete variables, all the variables in RHUC are regarded as continuous. Additionally, a series of updating regulations are defined to ensure that the model is realistic. Thus, a DRL algorithm, the twin delayed deep deterministic policy gradient (TD3), can be utilized to effectively solve the problem. Finally, several case studies are conducted based on different test systems to demonstrate the efficiency of the proposed method. According to the experimental results, the proposed algorithm can obtain high-quality solutions in a considerably shorter time than traditional methods, which leads to a reduction of at least 1.1% in the power system operation cost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Algorithm 2
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Venkateswaran R, Joo YH (2021) Retarded sampled-data control design for interconnected power system with dfig-based wind farm: Lmi approach. IEEE Trans Cybernet 52(7):5767–5777. https://doi.org/10.1109/TCYB.2020.3042543

    Article  Google Scholar 

  2. Yu R, Sun Y, Li X, Yu J, Gao J, Liu Z, Yu M (2022) Time series cross-correlation network for wind power prediction. Appl Intell

  3. Huang H, Jia R, Shi X, Liang J, Dang J (2021) Feature selection and hyper parameters optimization for short-term wind power forecast. Appl Intell 51(10):6752–6770. https://doi.org/10.1007/s10489-021-02191-y

    Article  Google Scholar 

  4. Zhang X (2022) Developing a hybrid probabilistic model for short-term wind speed forecasting. Appl Intell 53:728–745. https://doi.org/10.1007/s10489-022-03644-8

    Article  Google Scholar 

  5. Liu J, Wang J, Yu W, Wang Z, Zhong G, He F (2022) Semi-supervised deep learning recognition method for the new classes of faults in wind turbine system. Appl Intell 52(8):9212–9224. https://doi.org/10.1007/s10489-021-03024-8

    Article  Google Scholar 

  6. Li Y, Peng X, Zhang Y (2022) Forecasting methods for wind power scenarios of multiple wind farms based on spatio-temporal dependency structure. Renew Energy 201:950–960. https://doi.org/10.1016/j.renene.2022.11.002

    Article  Google Scholar 

  7. Wang Z, Wang W, Liu C, Wang B (2020) Forecasted scenarios of regional wind farms based on regular vine copulas. J Modern Power Syst Clean Energy 8(1):77–85. https://doi.org/10.35833/MPCE.2017.000570

    Article  Google Scholar 

  8. Reolon Scuzziato M, Cristian Finardi E, Frangioni A (2021) Solving stochastic hydrothermal unit commitment with a new primal recovery technique based on lagrangian solutions. Int J Electr Power Energy Syst 127:106661. https://doi.org/10.1016/j.ijepes.2020.106661

    Article  Google Scholar 

  9. Colonetti B, Finardi E, Larroyd P, Beltrán F (2022) A novel cooperative multi-search benders decomposition for solving the hydrothermal unit-commitment problem. Int J Electr Power Energy Syst 134:107390. https://doi.org/10.1016/j.ijepes.2021.107390

    Article  Google Scholar 

  10. Postolov B, Iliev A (2022) New metaheuristic methodology for solving security constrained hydrothermal unit commitment based on adaptive genetic algorithm. Int J Electr Power Energy Syst 134:107163. https://doi.org/10.1016/j.ijepes.2021.107163

    Article  Google Scholar 

  11. Wang B, Zhang P, He Y, Wang X, Zhang X (2022) Scenario-oriented hybrid particle swarm optimization algorithm for robust economic dispatch of power system with wind power. J Syst Eng Electr 33(5):1143–1150. https://doi.org/10.23919/JSEE.2022.000110

    Article  Google Scholar 

  12. Zhou M, Wang B, Watada J (2019) Deep learning-based rolling horizon unit commitment under hybrid uncertainties. Energy 186:115843. https://doi.org/10.1016/j.energy.2019.07.173

    Article  Google Scholar 

  13. Li Y, Xu Z, Wang X, Wang X (2020) A bibliometric analysis on deep learning during 2007-2019. Int J Mach Learn Cybernet 11(12):2807–2826

    Article  Google Scholar 

  14. CRISTESCU M-C (2021) Machine learning techniques for improving the performance metrics of functional verification. Rom J Inf Sci Technol 24(1):99–116

    MathSciNet  Google Scholar 

  15. Zamfirache IA, Precup R-E, Roman R-C, Petriu EM (2022) Policy iteration reinforcement learning-based control using a grey wolf optimizer algorithm. Inf Sci 585:162–175. https://doi.org/10.1016/j.ins.2021.11.051

    Article  Google Scholar 

  16. Wang D, Hu M, Weir JD (2022) Simultaneous task and energy planning using deep reinforcement learning. Inf Sci 607:931–946. https://doi.org/10.1016/j.ins.2022.06.015

    Article  Google Scholar 

  17. Qi C, Song C, Xiao F, Song S (2022) Generalization ability of hybrid electric vehicle energy management strategy based on reinforcement learning method. Energy 250:123826. https://doi.org/10.1016/j.energy.2022.123826

    Article  Google Scholar 

  18. Fang D, Guan X, Hu B, Peng Y, Chen M, Hwang K (2021) Deep reinforcement learning for scenario-based robust economic dispatch strategy in internet of energy. IEEE Internet Things J 8 (12):9654–9663. https://doi.org/10.1109/JIOT.2020.3040294

    Article  Google Scholar 

  19. Yan Z, Xu Y (2020) Real-time optimal power flow: A lagrangian based deep reinforcement learning approach. IEEE Trans Power Syst 35(4):3270–3273. https://doi.org/10.1109/TPWRS.2020.2987292

    Article  Google Scholar 

  20. Wu J, Wang J, Kong X (2022) Strategic bidding in a competitive electricity market: An intelligent method using multi-agent transfer learning based on reinforcement learning. Energy 256:124657. https://doi.org/10.1016/j.energy.2022.124657

    Article  Google Scholar 

  21. Yang Q, Wang G, Sadeghi A, Giannakis GB, Sun J (2020) Two-timescale voltage control in distribution grids using deep reinforcement learning. IEEE Trans Smart Grid 11(3):2313–2323. https://doi.org/10.1109/TSG.2019.2951769

    Article  Google Scholar 

  22. Li F, Qin J, Zheng WX (2020) Distributed q-learning-based online optimization algorithm for unit commitment and dispatch in smart grid. IEEE Trans Cybernet 50(9):4146–4156. https://doi.org/10.1109/TCYB.2019.2921475

    Article  Google Scholar 

  23. Liu W, Zhuang P, Liang H, Peng J, Huang Z (2018) Distributed economic dispatch in microgrids based on cooperative reinforcement learning. IEEE Trans Neural Netw Learn Syst 29(6):2192–2203. https://doi.org/10.1109/TNNLS.2018.2801880

    Article  MathSciNet  Google Scholar 

  24. Qin J, Yu N, Gao Y (2021) Solving unit commitment problems with multi-step deep reinforcement learning. In: 2021 IEEE international conference on communications, control, and computing technologies for smart grids (SmartGridComm), pp 140–145, DOI https://doi.org/10.1109/SmartGridComm51999.2021.9632339

  25. Zou J, Ahmed S, Sun XA (2019) Multistage stochastic unit commitment using stochastic dual dynamic integer programming. IEEE Trans Power Syst 34(3):1814–1823. https://doi.org/10.1109/TPWRS.2018.2880996

    Article  Google Scholar 

  26. Bhadoria A, Kamboj VK (2019) Optimal generation scheduling and dispatch of thermal generating units considering impact of wind penetration using hgwo-res algorithm. Appl Intell 49:1517–1547. https://doi.org/10.1007/s10489-018-1325-9

    Article  Google Scholar 

  27. Silvente J, Kopanos GM, Dua V, Papageorgiou LG (2018) A rolling horizon approach for optimal management of microgrids under stochastic uncertainty. Chem Eng Res Des 131:293–317. https://doi.org/10.1016/j.cherd.2017.09.013. Energy Systems Engineering

    Article  Google Scholar 

  28. Bakirtzis EA, Simoglou CK, Biskas PN, Bakirtzis AG (2018) Storage management by rolling stochastic unit commitment for high renewable energy penetration. Electr Power Syst Res 158:240–249. https://doi.org/10.1016/j.epsr.2017.12.025

    Article  Google Scholar 

  29. Dai P, Yu W, Wen G, Baldi S (2020) Distributed reinforcement learning algorithm for dynamic economic dispatch with unknown generation cost functions. IEEE Trans Ind Inf 16(4):2258–2267. https://doi.org/10.1109/TII.2019.2933443

    Article  Google Scholar 

  30. Li D, Yu L, Li N, Lewis F (2021) Virtual-action-based coordinated reinforcement learning for distributed economic dispatch. IEEE Trans Power Syst 36(6):5143–5152. https://doi.org/10.1109/TPWRS.2021.3070161

    Article  Google Scholar 

  31. Ajagekar A, You F (2022) Deep reinforcement learning based solution approach for unit commitment under demand and wind power uncertainty. In: 2022 american control conference (ACC), pp 4520–4525. https://doi.org/10.23919/ACC53348.2022.9867273

  32. Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2016) Continuous control with deep reinforcement learning. In: ICLR (Poster)

  33. Fujimoto S, van Hoof H, Meger D (2018) Addressing function approximation error in actor-critic methods. In: Proceedings of Machine Learning Research, vol. 80, ICML, pp 1582– 1591

  34. Hu J, Li H (2019) A new clustering approach for scenario reduction in multi-stochastic variable programming. IEEE Trans Power Syst 34(5):3813–3825. https://doi.org/10.1109/TPWRS.2019.2901545

    Article  Google Scholar 

  35. Tavakoli A, Karimi A, Shafie-khah M (2022) Stochastic optimal operation framework of an integrated methane-based zero-co2 energy hub in energy markets. Electr Power Syst Res 209:108005. https://doi.org/10.1016/j.epsr.2022.108005

    Article  Google Scholar 

  36. He Y, Wu H, Ding M, Bi R, Hua Y (2023) Reduction method for multi-period time series scenarios of wind power. Electr Power Syst Res 214:108813. https://doi.org/10.1016/j.epsr.2022.108813

    Article  Google Scholar 

  37. Huang D, Wang C, Wu J, Lai J, Kwoh CK (2020) Ultra-scalable spectral clustering and ensemble clustering. IEEE Trans Knowl Data Eng 32(6):1212–1226. https://doi.org/10.1109/TKDE.2019.2903410

    Article  Google Scholar 

  38. Chen Y, Wang Y, Kirschen D, Zhang B (2018) Model-free renewable scenario generation using generative adversarial networks. IEEE Trans Power Syst 33(3):3265–3275. https://doi.org/10.1109/TPWRS.2018.2794541

    Article  Google Scholar 

  39. Yuan R, Wang B, Sun Y, Song X, Watada J (2022) Conditional style-based generative adversarial networks for renewable scenario generation. IEEE Trans Power Syst :1–1

  40. Wang B, Wang S, Zhou X, Watada J (2016) Two-stage multi-objective unit commitment optimization under hybrid uncertainties. IEEE Trans Power Syst 31(3):2266–2277. https://doi.org/10.1109/TPWRS.2015.2463725

    Article  Google Scholar 

  41. Group E Transparency on grid data [DB/OL]. http://www.elia.be/en/grid-data

  42. Yilmaz ÖF, Yazici B (2022) Tactical level strategies for multi-objective disassembly line balancing problem with multi-manned stations: an optimization model and solution approaches. Annals Oper Res 319 (2):1793–1843

    Article  MATH  Google Scholar 

  43. Yilmaz OF, Ozcelik G, Yeni FB (2020) Lean holistic fuzzy methodology employing cross-functional worker teams for new product development projects: A real case study from high-tech industry. Eur J Oper Res 282(3):989–1010. https://doi.org/10.1016/j.ejor.2019.09.048

    Article  MathSciNet  MATH  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (Grant No. 61603176).

Author information

Author notes

    Authors and Affiliations

    1. School of Management and Engineering, Nanjing University, Nanjing, 210093, China

      Jinhao Shi, Bo Wang, Ran Yuan, Zhi Wang & Chunlin Chen

    2. Graduate School of Information, Production and Systems, Waseda University, Kitakyushu, 808-0135, Japan

      Junzo Watada

    Authors
    1. Jinhao Shi
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Bo Wang
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Ran Yuan
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Zhi Wang
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Chunlin Chen
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Junzo Watada
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Bo Wang.

    Ethics declarations

    Conflict of Interests

    The authors declare no conflict of interest.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Ran Yuan, Zhi Wang, Chunlin Chen and Junzo Watada are contributed equally to this work.

    Rights and permissions

    Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Shi, J., Wang, B., Yuan, R. et al. Rolling horizon wind-thermal unit commitment optimization based on deep reinforcement learning. Appl Intell 53, 19591–19609 (2023). https://doi.org/10.1007/s10489-023-04489-5

    Download citation

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s10489-023-04489-5

    Keywords

    Search

    Navigation