Skip to main content
SpringerLink
Log in

Capacity planning for integrated energy system based on reinforcement learning and multi-criteria evaluation

  • Original Paper
  • Published:
Energy Systems Aims and scope Submit manuscript

Abstract

Optimal capacity planning for energy devices is significantly crucial for saving economic costs and enhancing operational efficiency in an integrated energy system (IES). In this study, a reinforcement learning (RL)-based capacity planning approach for IES is proposed, where a multistage decision-making strategy is designed to reduce the action dimensionality for improving computational efficiency. Besides, to evaluate each capacity configuration scheme in RL process, a sound multi-criteria system from aspects of economy, environment, efficiency and safety is built. Within it, some new indicators are innovatively developed, including a capacity factor criterion, an installed capacity ratio and generation ratio of renewable energy devices as well as an adjust rate of energy generators. To verify the effectiveness of the proposed approach, a case study for an industrial park is carried out. The experimental results demonstrate that the proposed approach outperforms the conventional mixed integer linear programming (MILP) and multi-objective optimization (MOO) -based methods on planning the optimal capacity. Furthermore, comparative experiments with separate production systems are conducted to further study the advantages of IES, and sensitive analyses are also performed to verify the robustness of determining the optimal capacity configuration of this proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The data that support the findings of this study are available on request from the corresponding author, [Jun Zhao], upon reasonable request.

Abbreviations

IES:

integrated energy system

T:

transformer

PV:

transformer

WT:

wind turbine

GICE:

gas internal combustion engine

GB:

gas boiler

EHP:

electric heat pump

AC:

absorption chiller

EC:

electric chiller

PS:

power storage

HS:

heat storage

\(N_{d}\) :

number of typical scenarios

\(T_{d}\) :

time period in one day

\(D_{d}\) :

dth scenario duration

\(c_{c}^{e}\) :

electricity carbon emission cost

\(c_{c}^{ng}\) :

natural gas carbon emission cost

\(c_{e}\) :

electricity purchased cost

\(c_{ng}\) :

natural gas purchased cost

\(c_{g,m}\) :

maintenance cost of generation g

\(c_{s,m}\) :

maintenance cost of storage s

\(\eta_{g}^{k}\) :

efficiency of generation g for kth system

\(\varpi_{g,\max }^{k}\) :

maximum range coefficient of generation g

\(\varpi_{g,\min }^{k}\) :

minimum range coefficient of generation g

\(\varpi_{s,\max }^{k}\) :

maximum range coefficient of storage s

\(\varpi_{s,\min }^{k}\) :

minimum range coefficient of storage s

\(\overline{P}_{g}^{k}\) :

maximum output of generation g

\(\underline {P}_{g}^{k}\) :

minimum output of generation g

\(\overline{E}_{s}^{k}\) :

maximum energy of storage s

\(\underline {E}_{s}^{k}\) :

minimum energy of storage s

\(P_{dem}^{k} (t)\) :

k energy demand at the time t

\(V_{g}\) :

capacity variable of generation g

\(V_{s}\) :

capacity variable of storage s

\(P_{T}^{e} (t)\) :

Transformer ouput at the time t

\(P_{g}^{k} (t)\) :

output of generation g

\(E_{s}^{k} (t)\) :

stored k energy of storage s

\(P_{s,chg}^{k} (t)\) :

charging power of storage s

\(P_{s,dchg}^{k} (t)\) :

discharging power of storage s

\(\alpha_{s}^{k} (t)\) :

working mode of storage s

\({\mathbf{s}}\) :

state of RL

\({\mathbf{a}}\) :

action set

\({\mathbf{A}}\) :

action spaces

\({\mathbf{S}}\) :

state spaces

k :

energy sub-system index

g :

energy generation index

s :

energy storage index

d :

the typical scenario index

t :

time step

i :

step of RL

References

  1. Rifkin, J.: The third industrial revolution: how lateral power is transforming energy, the economy, and the world. Macmillan (2011)

  2. Zhang, C., Peng, K., Han, Y., Wang, L., Zeng, S.Q., Dong, W.J.: Key technologies and system development for regional integrated energy system. Energy Rep. 6, 374–379 (2020)

    Article  Google Scholar 

  3. Wu, J.Z., Yan, J.Y., Jia, H.J., Hatziargyriou, N., Djilali, N., Sun, H.B.: Integrated energy systems. Appl. Energy. 167, 155–157 (2016)

    Article  Google Scholar 

  4. Xiong, J.J., Sun, Y.H., Wang, J.X., Li, Z.P., Xu, Z., Zhai, S.W.: Multi-stage equipment optimal configuration of park-level integrated energy system considering flexible loads. Int. J. Electr. Power Energy Syst. 140, 108050 (2022)

    Article  Google Scholar 

  5. Xiang, Y., Cai, H.H., Gu, C.H., Shen, X.D.: Cost-benefit analysis of integrated energy system planning considering demand response. Energy. 192, 116632 (2020)

    Article  Google Scholar 

  6. Ma, T.F., Wu, J.Y., Hao, L.L., Lee, W.J., Yan, H.G., Li, D.Z.: The optimal structure planning and energy management strategies of smart multi energy systems. Energy 160, 122–141 (2018)

    Article  Google Scholar 

  7. Gu, W., Wang, J., Lu, S., Luo, Z., Wu, C.Y.: Optimal operation for integrated energy system considering thermal inertia of district heating network and buildings. Appl. Energy. 199, 234–246 (2017)

    Article  Google Scholar 

  8. Ding, Y.X., Xu, Q.S., Yang, B., Zhou, H., Lu, L., Shen, L., Zhang, H.Y., Jiang, L., Liao, K.: Integrated location and capacity coordination planning scheme for multi-power complementary generation system. Energy Rep. 8, 10–18 (2022)

    Article  Google Scholar 

  9. Liu, Z.F., Chen, Y.X., Zhuo, R.Q., Jia, H.J.: Energy storage capacity optimization for autonomy microgrid considering CHP and EV scheduling. Appl. Energy. 210, 1113–1125 (2018)

    Article  Google Scholar 

  10. Wu, Q., Ren, H.B., Gao, W.J., Ren, J.X.: Modeling and optimization of distributed energy supply network with power and hot water interchanges. Appl. Therm. Eng. 94, 635–664 (2016)

    Article  Google Scholar 

  11. Wang, Y.L., Wang, Y.D., Huang, Y.J., Li, F., Zeng, M., Li, J.P., Wang, X.H., Zhang, F.W.: Planning and operation method of the regional integrated energy system considering economy and environment. Energy 171, 731–750 (2019)

    Article  Google Scholar 

  12. Sun, J.Q., Ruze, N., Zhang, J.J., Shi, J., Shen, B.Y.: Capacity planning and optimization for integrated energy system in industrial park considering environmental externalities. Renew. Energy. 167, 56–65 (2021)

    Article  Google Scholar 

  13. Khiareddine, A., Salah, C.B., Rekioua, D., Mimouni, M.F.: Sizing methodology for hybrid photovoltaic /wind/ hydrogen/battery integrated to energy management strategy for pumping system. Energy 153, 743–762 (2018)

    Article  Google Scholar 

  14. He, S.J., Gao, H.J., Wang, L.F., Xiang, Y.M., Liu, J.Y.: Distributionally robust planning for integrated energy systems incorporating electric-thermal demand response. Energy. 213, 1187 (2020)

    Article  Google Scholar 

  15. Yan, R.J., Wang, J.J., Lu, S.K., Ma, Z.R., Zhou, Y., Zhang, L.D., Cheng, Y.L.: Multi-objective two-stage adaptive robust planning method for an integrated energy system considering load uncertainty. Energy Build. 235, 110741 (2021)

    Article  Google Scholar 

  16. Bhattacharjee, S., Sioshansi, R., Zareipour, H.: Benefits of strategically sizing wind-integrated energy storage and transmission. IEEE Trans. Power Syst. 36, 1141–1151 (2020)

    Article  Google Scholar 

  17. Yousefi, H., Ghodusinejad, M.H., Noorollahi, Y.: GA/AHP-based optimal design of a hybrid CCHP system considering economy, energy and emission. Energy Build. 138, 309–317 (2017)

    Article  Google Scholar 

  18. Lorestani, A., Ardehali, M.M.: Optimal integration of renewable energy sources for autonomous tri-generation combined cooling, heating and power system based on evolutionary particle swarm optimization algorithm. Energy 145, 839–855 (2018)

    Article  Google Scholar 

  19. Guo, L., Liu, W.J., Cai, J.J., Hong, B.W., Wang, C.S.: A two-stage optimal planning and design method for combined cooling, heat and power microgrid system. Energy Conv. Manag. 74, 433–445 (2013)

    Article  Google Scholar 

  20. Zhang, L., Zhou, P., Newton, S., Fang, J.X., Zhou, D.Q., Zhang, L.P.: Evaluating clean energy alternatives for Jiangsu, China: an improved multi-criteria decision making method. Energy 90(1), 953–964 (2015)

    Article  Google Scholar 

  21. Çelikbilek, Y., Tüysüz, F.: An integrated grey based multi-criteria decision making approach for the evaluation of renewable energy sources. Energy 115(1), 1246–1258 (2016)

    Article  Google Scholar 

  22. Karatas, M., Sulukan, E., Karacan, I.: Assessment of Turkey’s energy management performance via a hybrid multi-criteria decision-making methodology. Energy 153, 890–912 (2018)

    Article  Google Scholar 

  23. Supciller, A.A., Toprak, F.: Selection of wind turbines with multi-criteria decision making techniques involving neutrosophic numbers: a case from Turkey. Energy 207, 118237 (2020)

    Article  Google Scholar 

  24. Muhsen, D.H., Khatib, T., Abdulabbas, T.E.: Sizing of a standalone photovoltaic water pumping system using hybrid multi-criteria decision making methods. Sol. Energy 159, 1003–1015 (2018)

    Article  Google Scholar 

  25. Hajibandeh, N., Shafie-khah, M., Osório, G.J., Aghaei, J., Catalão, J.P.S.: A heuristic multi-objective multi-criteria demand response planning in a system with high penetration of wind power generators. Appl. Energy. 212, 721–732 (2018)

    Article  Google Scholar 

  26. Ebrahimi, M., Keshavarz, A.: Sizing the prime mover of a residential micro-combined cooling heating and power (CCHP) system by multi-criteria sizing method for different climates. Energy 54, 291–301 (2013)

    Article  Google Scholar 

  27. Yang, K., Ding, Y., Zhu, N., Yang, F., Wang, Q.C.: Multi-criteria integrated evaluation of distributed energy system for community energy planning based on improved grey incidence approach: a case study in Tianjin. Appl. Energy. 229, 352–363 (2018)

    Article  Google Scholar 

  28. Hua, H., Qin, Y.C., Hao, C.T., Cao, J.W.: Optimal energy management strategies for energy Internet via deep reinforcement learning approach. Appl. Energy. 239, 598–609 (2019)

    Article  Google Scholar 

  29. Han, X.F., He, H.W., Wu, J.D., Peng, J.K., Li, Y.C.: Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle. Appl. Energy. 254, 113708 (2019)

    Article  Google Scholar 

  30. Yousefi, N., Tsianikas, S., Coit, D.W.: Reinforcement learning for dynamic condition-based maintenance of a system with individually repairable components. Qual. Eng. 32, 388–408 (2020)

    Article  Google Scholar 

  31. Liu, W.R., Zhuang, P., Liang, H., Peng, J., Huang, Z.W.: Distributed economic dispatch in microgrids based on cooperative reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2192–2203 (2018)

    Article  MathSciNet  Google Scholar 

  32. Foruzan, E., Soh, L.K., Asgarpoor, S.: Reinforcement learning approach for optimal distributed energy management in a microgrid. IEEE Trans. Power Syst. 33(5), 5749–5758 (2018)

    Article  Google Scholar 

  33. Ebell, N., Heinrich, F., Schlund, J., Pruckner, M.: Reinforcement Learning Control Algorithm for a PV-Battery-System Providing Frequency Containment Reserve Power, 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), Aalborg, Denmark (2018)

  34. Tuchnitz, F., Ebell, N., Schlund, J., Pruckner, M.: Development and evaluation of a smart charging strategy for an electric vehicle fleet based on reinforcement learning. Appl. Energy. 285, 116382 (2021)

    Article  Google Scholar 

  35. Du, G.D., Zou, Y., Zhang, X.D., Liu, T., Wu, J.L., He, D.B.: Deep reinforcement learning based energy management for a hybrid electric vehicle. Energy 201, 117591 (2020)

    Article  Google Scholar 

  36. Wu, J.D., Wei, Z.B., Li, W.H., Wang, Y., Li, Y.W., Sauer, D.U.: Battery thermal- and health-constrained energy management for hybrid electric bus based on soft actor-critic DRL algorithm. IEEE Trans. Ind. Inform. 17(6), 3751–3761 (2021)

    Article  Google Scholar 

  37. Wu, J.D., Wei, Z.B., Liu, K.L., Quan, Z.Y., Li, Y.W.: Battery-involved energy management for hybrid electric bus based on expert-assistance deep deterministic policy gradient algorithm. IEEE Trans. Veh. Technol. 69(11), 12786–12796 (2020)

    Article  Google Scholar 

  38. Tsianikas, S., Yousefi, N., Yang, J., Rodgers, M., Coit, D.W.: A Sequential Resource Investment Planning Framework using Reinforcement Learning and Simulation-Based Optimization. Appl. Energy (2021).

  39. Perera, A.T.D., Wickramasinghe, P.U., Nik, V.M., Scartezzini, J.L.: Introducing reinforcement learning to the energy system design process. Appl. Energy. 262, 114580 (2020)

    Article  Google Scholar 

  40. Çolak, M., Kaya, I.: Prioritization of renewable energy alternatives by using an integrated fuzzy MCDM model: a real case application for Turkey. Renew. Sust. Energ. Rev. 80, 840–853 (2017)

    Article  Google Scholar 

  41. Yang, Y., Zhang, S.J., Xiao, Y.H.: Optimal design of distributed energy resource systems coupled with energy distribution networks. Energy 85, 433–448 (2015)

    Article  Google Scholar 

  42. Mi, X.M., Liu, R., Cui, H.Z., Memon, S.A., Xing, F., Lo, Y.: Energy and economic analysis of building integrated with PCM in different cities of China. Appl. Energy. 175, 324–336 (2016)

    Article  Google Scholar 

  43. Stein, E.W.: A comprehensive multi-criteria model to rank electric energy production technologies. Renew. Sust. Energ. Rev. 22, 640–654 (2013)

    Article  Google Scholar 

  44. Yang, H.M., Xiong, T.L., Qiu, J., Qiu, D., Dong, Y.Z.: Optimal operation of DES/CCHP based regional multi-energy prosumer with demand response. Appl. Energy. 167, 353–365 (2016)

    Article  Google Scholar 

  45. Lofberg, J.: YALMIP: a toolbox for modeling and optimization in MATLAB, IEEE International Symposium on Computer Aided Control Systems Design. IEEE (2005).

  46. Yang, X.H., Chen, Z.X., Huang, X., Li, R.X., Xu, S.P., Yang, C.S.: Robust capacity optimization methods for integrated energy systems considering demand response and thermal comfort. Energy 221, 119727 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Key R&D Program of China under Grant 2017YFA0700300, the National Natural Sciences Foundation of China under Grant 61833003, Grant 61533005, Grant U1908218, 62003072, and the Outstanding Youth Sci-Tech Talent Program of Dalian under Grant 2018RJ01.

Author information

Authors and Affiliations

  1. Key Laboratory of Intelligent Control and Optimization for Industrial Equipment, Ministry of Education, Dalian University of Technology, Dalian, 116024, China

    Fan Zhou, Long Chen, Jun Zhao & Wei Wang

  2. School of Control Science and Engineering, Dalian University of Technology, Dalian, 116024, China

    Fan Zhou, Long Chen, Jun Zhao & Wei Wang

Authors
  1. Fan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Long Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Zhao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, F., Chen, L., Zhao, J. et al. Capacity planning for integrated energy system based on reinforcement learning and multi-criteria evaluation. Energy Syst (2023). https://doi.org/10.1007/s12667-023-00603-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s12667-023-00603-1

Keywords

Search

Navigation