Skip to main content
SpringerLink
Log in

Deep Reinforcement Learning for Optimal Energy Management of Multi-energy Smart Grids

  • Conference paper
  • First Online:
Machine Learning, Optimization, and Data Science (LOD 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 13164))

Abstract

This paper proposes a Deep Reinforcement Learning approach for optimally managing multi-energy systems in smart grids. The optimal control problem of the production and storage units within the smart grid is formulated as a Partially Observable Markov Decision Process (POMDP), and is solved using an actor-critic Deep Reinforcement Learning algorithm. The framework is tested on a novel multi-energy residential microgrid model that encompasses electrical, heating and cooling storage as well as thermal production systems and renewable energy generation. One of the main challenges faced when dealing with real-time optimal control of such multi-energy systems is the need to take multiple continuous actions simultaneously. The proposed Deep Deterministic Policy Gradient (DDPG) agent has shown to handle well the continuous state and action spaces and learned to simultaneously take multiple actions on the production and storage systems that allow to jointly optimize the electrical, heating and cooling usages within the smart grid. This allows the approach to be applied for the real-time optimal energy management of larger scale multi-energy Smart Grids like eco-distrits and smart cities where multiple continuous actions need to be taken simultaneously.

Supported by the program Investissement d’Avenir, operated by l’Agence de l’Environnement et de la Maitrise de l’Energie ADEME, France.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Abbreviations

\(P_\text {Grid}\) :

Grid power consumption

\(P_\text {gen}\) :

Distributed power generation

\(C_\text {Grid}\) :

Cost of power purchase from the grid

\(C_\text {gen}\) :

Cost of distributed power generation

\(P_\text {Load}\) :

Load power

\(P_\text {pv}\) :

PV power generation

\(P_\text {Bat}\) :

Battery power

\(P_\text {H2}\) :

Hydrogen storage power

\(P_\text {TRHP}\) :

Electric power consumed by TRHP

\(Q_{\text {TRHP},t}^{H-prod}\) :

Heat produced by TRHP

\(Q_{\text {TRHP},t}^{C-prod}\) :

Cold produced by TRHP

\(COP_{TRHP}\) :

Coefficient of performance of TRHP

\(Q_{H-load}\) :

Heating load

\(Q_{C-load}\) :

Cooling load

t :

Time step

\(P_\text {(i)}\) :

Power of a storage system i

\(P_\text {Ch}^{(i)}\) :

Charging power of a storage system i

\(P_\text {Disch}^{(i)}\) :

Discharging power of a storage system i

\(P_\text {min}^{(i)}\) :

Minimum power of storage system i

\(P_\text {max}^{(i)}\) :

Maximum power of storage system i

\(\eta _\text {Ch}^{(i)}\) :

Charging efficiency of a storage system i

\(\eta _\text {Disch}^{(i)}\) :

Discharging efficiency of a storage system i

\(k_\text {sd}^{(i)}\) :

Self-discharge rate of a storage system i

\(E_{init}^{(i)}\) :

Energy initially stored in storage system i

\(E^{(i)}\) :

Energy stored in storage system i

PV :

Photo-voltaic

SoC :

State of Charge

MG :

Microgrid

SG :

Smart Grid

TRHP :

Thermo-Refrigerating Heat Pump

SDHS :

Smart District Heating System

MPC :

Model Predictive Control

MDP :

Markov Decision Process

ML :

Machine Learning

DL :

Deep Learning

RL :

Reinforcement Learning

DRL :

Deep Reinforcement Learning

DQN :

Deep Q-Networks

DQL :

Deep Q-Learning

DPG :

Deep Policy Gradient

DDPG :

Deep Deterministic Policy Gradient

References

  1. Bousnina, D., de Oliveira, W., Pflaum, P.: A stochastic optimization model for frequency control and energy management in a microgrid. In: Nicosia, G., et al. (eds.) LOD 2020. LNCS, vol. 12565, pp. 177–189. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64583-0_17

    Chapter  Google Scholar 

  2. de Bruin, T., Kober, J., Tuyls, K., Babuška, R.: Improved deep reinforcement learning for robotics through distribution-based experience retention. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3947–3952. IEEE (2016)

    Google Scholar 

  3. Carta, S., Ferreira, A., Podda, A.S., Recupero, D.R., Sanna, A.: Multi-DQN: An ensemble of deep q-learning agents for stock market forecasting. Expert Syst. Appl. 164, 113820 (2021)

    Article  Google Scholar 

  4. van den Ende, M., Lukszo, Z., Herder, P.M.: Smart thermal grid. In: 2015 IEEE 12th International Conference on Networking, Sensing and Control, pp. 432–437. IEEE (2015)

    Google Scholar 

  5. Fang, X., Misra, S., Xue, G., Yang, D.: Smart grid-the new and improved power grid: a survey. IEEE Commun. Surv. Tutorials 14(4), 944–980 (2011)

    Article  Google Scholar 

  6. François-Lavet, V.: Contributions to deep reinforcement learning and its applications in smartgrids. Ph.D. thesis, Université de Liège, Liège, Belgique (2017)

    Google Scholar 

  7. François-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G., Pineau, J.: An introduction to deep reinforcement learning. arXiv preprint arXiv:1811.12560 (2018)

  8. François-Lavet, V., Taralla, D., Ernst, D., Fonteneau, R.: Deep reinforcement learning solutions for energy microgrids management. In: European Workshop on Reinforcement Learning (EWRL 2016) (2016)

    Google Scholar 

  9. Gao, G., Li, J., Wen, Y.: Energy-efficient thermal comfort control in smart buildings via deep reinforcement learning. arXiv preprint arXiv:1901.04693 (2019)

  10. Garcia, C.E., Prett, D.M., Morari, M.: Model predictive control: theory and practice-a survey. Automatica 25(3), 335–348 (1989)

    Article  MATH  Google Scholar 

  11. Gelleschus, R., Böttiger, M., Stange, P., Bocklisch, T.: Comparison of optimization solvers in the model predictive control of a PV-battery-heat pump system. Energ. Procedia 155, 524–535 (2018)

    Article  Google Scholar 

  12. Ji, Y., Wang, J., Xu, J., Fang, X., Zhang, H.: Real-time energy management of a microgrid using deep reinforcement learning. Energies 12(12), 2291 (2019)

    Article  Google Scholar 

  13. Knight, W.: Google just gave control over data center cooling to an AI (2018)

    Google Scholar 

  14. Konda, V.R., Tsitsiklis, J.N.: Actor-critic algorithms. In: Advances in Neural Information Processing Systems, pp. 1008–1014 (2000)

    Google Scholar 

  15. Kuang, Y., Wang, X., Zhao, H., Huang, Y., Chen, X., Wang, X.: Agent-based energy sharing mechanism using deep deterministic policy gradient algorithm. Energies 13(19), 5027 (2020)

    Article  Google Scholar 

  16. Liaw, R., Krishnan, S., Garg, A., Crankshaw, D., Gonzalez, J.E., Goldberg, K.: Composing meta-policies for autonomous driving using hierarchical deep reinforcement learning. arXiv preprint arXiv:1711.01503 (2017)

  17. Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)

  18. Lin, S., Yu, H., Chen, H.: On-line optimization of microgrid operating cost based on deep reinforcement learning. In: IOP Conference Series: Earth and Environmental Science, vol. 701, p. 012084. IOP Publishing (2021)

    Google Scholar 

  19. Liu, H., Yu, C., Wu, H., Duan, Z., Yan, G.: A new hybrid ensemble deep reinforcement learning model for wind speed short term forecasting. Energy 202, 117794 (2020)

    Article  Google Scholar 

  20. Lopez-Martin, M., Carro, B., Sanchez-Esguevillas, A.: Application of deep reinforcement learning to intrusion detection for supervised problems. Expert Syst. Appl. 141, 112963 (2020)

    Article  Google Scholar 

  21. Ma, T., Wu, J., Hao, L., Lee, W.J., Yan, H., Li, D.: The optimal structure planning and energy management strategies of smart multi energy systems. Energy 160, 122–141 (2018)

    Article  Google Scholar 

  22. Mancarella, P.: Smart multi-energy grids: concepts, benefits and challenges. In: 2012 IEEE Power and Energy Society General Meeting, pp. 1–2. IEEE (2012)

    Google Scholar 

  23. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Google Scholar 

  24. Mocanu, E., et al.: On-line building energy optimization using deep reinforcement learning. IEEE Trans. Smart Grid 10(4), 3698–3708 (2018)

    Article  Google Scholar 

  25. Morari, M., Lee, J.H.: Model predictive control: past, present and future. Comput. Chem. Eng. 23(4–5), 667–682 (1999)

    Article  Google Scholar 

  26. Parisio, A., Rikos, E., Glielmo, L.: A model predictive control approach to microgrid operation optimization. IEEE Trans. Control Syst. Technol. 22(5), 1813–1827 (2014)

    Article  Google Scholar 

  27. Pflaum, P., Alamir, M., Lamoudi, M.Y.: Comparison of a primal and a dual decomposition for distributed MPC in smart districts. In: 2014 IEEE International Conference on Smart Grid Communications (SmartGridComm), pp. 55–60. IEEE (2014)

    Google Scholar 

  28. Qin, J., Han, X., Liu, G., Wang, S., Li, W., Jiang, Z.: Wind and storage cooperative scheduling strategy based on deep reinforcement learning algorithm. In: Journal of Physics: Conference Series, vol. 1213, p. 032002. IOP Publishing (2019)

    Google Scholar 

  29. Sallab, A.E., Abdou, M., Perot, E., Yogamani, S.: Deep reinforcement learning framework for autonomous driving. Electron. Imaging 2017(19), 70–76 (2017)

    Article  Google Scholar 

  30. Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)

    Google Scholar 

  31. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M.: Deterministic policy gradient algorithms. In: International Conference on Machine Learning, pp. 387–395. PMLR (2014)

    Google Scholar 

  32. Sogabe, T., et al.: Smart grid optimization by deep reinforcement learning over discrete and continuous action space. In: 2018 IEEE 7th World Conference on Photovoltaic Energy Conversion (WCPEC)(A Joint Conference of 45th IEEE PVSC, 28th PVSEC & 34th EU PVSEC), pp. 3794–3796. IEEE (2018)

    Google Scholar 

  33. Stănişteanu, C.: Smart thermal grids-a review. The Scientific Bulletin of Electrical Engineering Faculty 1(ahead-of-print) (2017)

    Google Scholar 

  34. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)

    Google Scholar 

  35. Sutton, R.S., Barto, A.G., et al.: Introduction to Reinforcement Learning, vol. 135. MIT Press, Cambridge (1998)

    Google Scholar 

  36. Tuballa, M.L., Abundo, M.L.: A review of the development of smart grid technologies. Renew. Sustain. Energ. Rev. 59, 710–725 (2016)

    Article  Google Scholar 

  37. Vecerik, M., et al.: Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 (2017)

  38. Wang, T., Kamath, H., Willard, S.: Control and optimization of grid-tied photovoltaic storage systems using model predictive control. IEEE Trans. Smart Grid 5(2), 1010–1017 (2014)

    Article  Google Scholar 

  39. Yang, L., Entchev, E., Rosato, A., Sibilio, S.: Smart thermal grid with integration of distributed and centralized solar energy systems. Energy 122, 471–481 (2017)

    Article  Google Scholar 

  40. Yu, L., et al.: Deep reinforcement learning for smart home energy management. IEEE Internet Things J. 7(4), 2751–2762 (2019)

    Article  Google Scholar 

  41. Zhang, B., Hu, W., Cao, D., Huang, Q., Chen, Z., Blaabjerg, F.: Deep reinforcement learning-based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy. Energ. Convers. Manage. 202, 112199 (2019)

    Article  Google Scholar 

  42. Zhang, T., Luo, J., Chen, P., Liu, J.: Flow rate control in smart district heating systems using deep reinforcement learning. arXiv preprint arXiv:1912.05313 (2019)

Download references

Author information

Authors and Affiliations

  1. MINES ParisTech, PSL Research University, CMA - Centre de Mathématiques Appliquées, Sophia Antipolis, France

    Dhekra Bousnina & Gilles Guerassimoff

Authors
  1. Dhekra Bousnina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gilles Guerassimoff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dhekra Bousnina .

Editor information

Editors and Affiliations

  1. University of Catania, Catania, Italy

    Giuseppe Nicosia

  2. Department of Computer Science, University of Reading, Reading, UK

    Varun Ojha

  3. Department of Computer Science, University of Oxford, Oxford, UK

    Emanuele La Malfa

  4. Cambridge Judge Business School, University of Cambridge, Cambridge, UK

    Gabriele La Malfa

  5. Department of Biochemistry, University of Cambridge, Cambridge, UK

    Giorgio Jansen

  6. Department of Industrial and Systems Engineering, University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

  7. University of Catania, Catania, Italy

    Giovanni Giuffrida

  8. Department of Informatics, Dana-Farber Cancer Institute, Boston, MA, USA

    Renato Umeton

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bousnina, D., Guerassimoff, G. (2022). Deep Reinforcement Learning for Optimal Energy Management of Multi-energy Smart Grids. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2021. Lecture Notes in Computer Science(), vol 13164. Springer, Cham. https://doi.org/10.1007/978-3-030-95470-3_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-95470-3_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-95469-7

  • Online ISBN: 978-3-030-95470-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation