Decentralized Control of DR Using a Multi-agent Method

  • Soroush Najafi
  • Saber Talari
  • Amin Shokri Gazafroudi
  • Miadreza Shafie-khahEmail author
  • Juan Manuel Corchado
  • João P. S. Catalão
Part of the Studies in Systems, Decision and Control book series (SSDC, volume 145)


Demand response (DR) is one of the most cost-effective elements of residential and small industrial building for the purpose of reducing the cost of energy. Today with broadening of the smart grid, electricity market and especially smart home, using DR can reduce cost and even make profits for consumers. On the other hand, utilizing centralized controls and have bidirectional communications Bi-directional communication between DR aggregators and consumers make many problems such as scalability and privacy violation. In this chapter, we propose a multi-agent method based on a Q-learning algorithm Q-learning algorithm for decentralized control of DR. Q-learning is a model-free reinforcement learning Reinforcement learning technique and a simple way for agents to learn how to act optimally in controlled Markovian domains. With this method, each consumer adapts its bidding and buying strategy over time according to the market outcomes. We consider energy supply for consumers such as small-scale renewable energy generators. We compare the result of the proposed method with a centralized aggregator-based approach that shows the effectiveness of the proposed decentralized DR market Decentralized DR market.


Demand response Multi-agents Q-learning algorithm 



The work of Saber Talari, Miadreza Shafie-khah and João P.S. Catalão was supported by FEDER funds through COMPETE 2020 and by Portuguese funds through FCT, under Projects SAICT-PAC/0004/2015—POCI-01-0145-FEDER-016434, POCI-01-0145-FEDER-006961, UID/EEA/50014/2013, UID/CEC/50021/2013, and UID/EMS/00151/2013. Also, the research leading to these results has received funding from the EU Seventh Framework Programme FP7/2007-2013 under grant agreement no. 309048.

Amin Shokri Gazafroudi and Juan Manuel Corchado acknowledge the support by the European Commission H2020 MSCA-RISE-2014: Marie Sklodowska-Curie project DREAM-GO Enabling Demand Response for short and real-time Efficient And Market Based Smart Grid Operation—An intelligent and real-time simulation approach ref. 641794. Moreover, Amin Shokri Gazafroudi acknowledge the support by the Ministry of Education of the Junta de Castilla y León and the European Social Fund through a grant from predoctoral recruitment of research personnel associated with the research project “Arquitectura multiagente para la gestión eficaz de redes de energía a través del uso de técnicas de intelligencia artificial” of the University of Salamanca.


  1. 1.
    Assessment of demand response and advanced metering. Washington, DC, USA, Tech. Rep., Dec. 2012Google Scholar
  2. 2.
    Y. Li, B.L. Ng, M. Trayer, L. Liu, Automated residential demand response: Algorithmic implications of pricing models. IEEE Trans. Smart Grid 3(4), 1712–1721 (2012)CrossRefGoogle Scholar
  3. 3.
    H. Aalami,M.P. Moghadam, G R. Yousefi, Optimum time of use program proposal for Iranian power systems, in Proceedings International Conference on Electrical Power Energy Conversion Systems, November 2009Google Scholar
  4. 4.
    S. Ashok, R. Banerjee, Optimal operation of industrial cogeneration for load management. IEEE Transactions Power System 18(2), 931–937 (2003)CrossRefGoogle Scholar
  5. 5.
    J. Joo,S. Ahn, Y. Yoon,J. Choi, Option valuation applied to implementing demand response via critical peak pricing, in Proceedings IEEE Power Energy Society General Meeting, Jun 2007Google Scholar
  6. 6.
    A.B. Philpott, E. Pettersen, Optimizing demand-side bids in day-ahead electricity market. IEEE Trans. Power Syst. 21(2), 488–498 (2006)CrossRefGoogle Scholar
  7. 7.
  8. 8.
    M. Shafie-khah et al., Optimal behavior of responsive residential demand considering hybrid phase change materials. Appl. Energy 163, 81–92 (2016)CrossRefGoogle Scholar
  9. 9.
    F. Wang et al., The values of market-based demand response on improving power system reliability under extreme circumstances. Appl Energy 193 220–231 (2017)Google Scholar
  10. 10.
    Q. Chen et al., Dynamic Price vector formation model based automatic demand response strategy for PV-assisted EV charging station. IEEE Trans. Smart Grid (2017)Google Scholar
  11. 11.
    F. Kamyab et al., Demand response program in smart grid using supply function bidding mechanism. IEEE Trans. Smart Grid 7(3), 1277–1284 (2016)Google Scholar
  12. 12.
    M.G. Vayá, L B. Roselló, G. Andersson, Optimal bidding of plug-in electric vehicles in a market-based control setup, in Power Systems Computation Conference (2014)Google Scholar
  13. 13.
    J. Mohammadi, G. Hug, S. Kar, Agent-based distributed security constrained optimal power flow, in IEEE Transactions on Smart Grid (2016)Google Scholar
  14. 14.
    S. Bahrami, M.H. Amini, A decentralized framework for real-time energy trading in distribution networks with load and generation uncertainty (2017), arXiv:1705.02575
  15. 15.
    M.H. Amini, B. Nabi, M.-R. Haghifam, Load management using multi-agent systems in smart distribution network, in IEEE Power and Energy Society General Meeting (PES) (IEEE, 2013)Google Scholar
  16. 16.
    M.G. Vayá, Roselló, G. Andersson, Centralized and decentralized approaches to smart charging of plug-in vehicles, in IEEE Power and Energy Society General Meeting (2012)Google Scholar
  17. 17.
    N. Rotering, M. Ilic, Optimal charge control of plug-in hybrid electric vehicles in deregulated electricity markets. IEEE Trans. Power Syst. 26(3), 1021–1029 (2011)CrossRefGoogle Scholar
  18. 18.
    S. Bashash, S.J. Moura, J.C. Forman, H.K. Fathy, Plug-in hybrid electric vehicle charge pattern optimization for energy cost and battery longevity. J. Power Sources 196(1), 541–549 (2011)CrossRefGoogle Scholar
  19. 19.
    A. Hoke, A. Brissette, D. Maksimovic, A. Pratt, K. Smith, Electric vehicle charge optimization including effects of lithium-ion battery degradation, in IEEE vehicle power and propulsion conference (2011) Google Scholar
  20. 20.
    J.K. Kok, M.J.J. Scheepers, I.G. Kamphuis, Intelligence in electricity networks for embedding renewables and distributed generations, Intelligent Infrastructures, Intelligent Systems, Control and Automation: Science and Engineering, vol. 42 (Springer, Netherlands, 2010), pp. 179–209zbMATHGoogle Scholar
  21. 21.
    Z. Ma, D.S. Callaway, I.A. Hiskens, Decentralized charging control of large populations of plug-in electric vehicles. IEEE Trans. Control Syst. Technol. 21(1), 67–78 (2013)CrossRefGoogle Scholar
  22. 22.
    L. Gan, U. Topcu, S. Low, Optimal decentralized protocol for electric vehicle charging. IEEE Trans. Power Syst. 28(2), 940–951 (2013)CrossRefGoogle Scholar
  23. 23.
    S. Bahrami, M.H. Amini, M. Shafie-khah, J.P.S. Catalao, A decentralized electricity market scheme enabling demand response deployment. IEEE Trans. Power Syst. (2017). Google Scholar
  24. 24.
    L.P. Kaelbling, M.L. Littman, A.W. Moore, Reinforcement learning: a survey. J. Artif. Int. Res. 4, 237–285 (1996)Google Scholar
  25. 25.
    C.J.C.H. Watkins, Learning from delayed rewards. Ph.D. thesis, King’s College, Cambridge, 1989Google Scholar
  26. 26.
    T. Krause, et al., A comparison of Nash equilibria analysis and agent-based modelling for power markets. Int. J. Electr. Power Energy Syst. 28(9), 599–607 (2006)Google Scholar
  27. 27.
    H.T. Haider, O.H. See, W. Elmenreich, Residential demand response scheme based on adaptative consumption level pricing. Energy 113, 301–308 (2016)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Soroush Najafi
    • 1
  • Saber Talari
    • 2
  • Amin Shokri Gazafroudi
    • 3
  • Miadreza Shafie-khah
    • 2
    Email author
  • Juan Manuel Corchado
    • 3
  • João P. S. Catalão
    • 2
    • 4
    • 5
  1. 1.Department of Electrical EngineeringIsfahan University of TechnologyIsfahanIran
  2. 2.C-MASTUniversity of Beira InteriorCovilhãPortugal
  3. 3.BISITE Research GroupUniversity of SalamancaSalamancaSpain
  4. 4.INESC TEC and the Faculty of Engineering of the University of PortoPortoPortugal
  5. 5.INESC-ID, Instituto Superior TécnicoUniversity of LisbonLisbonPortugal

Personalised recommendations