Skip to main content

Advertisement

SpringerLink
Log in

Solving dynamic distribution network reconfiguration using deep reinforcement learning

  • Original Paper
  • Published:
Electrical Engineering Aims and scope Submit manuscript

Abstract

Distribution network reconfiguration, as a part of the distribution management system, plays an important role in increasing the energy efficiency of the distribution network by coordinating the operations of the switches in the distribution network. Dynamic distribution network reconfiguration (DDNR), enabled by the sufficient number of remote switching devices in the distribution network, attempts to find the optimal topologies of the distribution network over the specified time interval. This paper proposes data-driven DDNR based on deep reinforcement learning (DRL). DRL-based DDNR controller aims to minimize the objective function, i.e. active energy losses and the cost of switching manipulations while satisfying the constraints. The following constraints are considered: allowed bus voltages, allowed line apparent powers, a radial network configuration with all buses being supplied, and the maximal allowed number of switching operations. This optimization problem is modelled as a Markov decision process by defining the possible states and actions of the DDNR agent (controller) and rewards that lead the agent to minimize the objective function while satisfying the constraints. Switching operation constraints are modelled by modifying the action space definition instead of including the additional penalty term in the reward function, to increase the computational efficiency. The proposed algorithm was tested on three test examples: small benchmark network, real-life large-scale test system and IEEE 33-bus radial system, and the results confirmed the robustness and scalability of the proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. Costs of energy losses and switching manipulations are the same as in Sect. 5.1.

  2. Costs of energy losses and switching manipulations are the same as in Sect. 5.1.

Abbreviations

\(T_{{{\text{int}}}}^{i}\) :

Duration of the considered time interval, in hours,

\(P,~Q\) :

Active and reactive powers, respectively

\(S^{{{\text{sw}}}}\) :

Apparent power of switch sw (sw =1, 2, … , Nsw, where NSW is the number of switches in the network)

\(P_{{{\text{Loss}}}}\) :

Active power losses

\(U,\theta\) :

Bus voltage magnitude and phase angle, respectively

\(x\) :

Switch status (1 – closed switch, 0 – opened switch)

\(y\) :

Change of switch status (1 – status change, 0 – no status change)

\(\alpha\) :

Auxiliary variable

\(p\left( {s^{\prime } ,~r|s,~a} \right)\) :

Transition function

\(S_{t} ,~s^{\prime } ,s_{t}\) :

Random variable denoting the new state in timestep t and its particular values, respectively

\(S_{{t - 1}} ,~s,~s_{{t - 1}}\) :

Random variable denoting the previous state in timestep t − 1 and its particular values, respectively

\(R_{t} ,~r,~r_{t}\) :

Random variable denoting the immediate reward in timestep t and its particular values, respectively

\(A_{{t - 1}} ,~a,~a_{{t - 1}}\) :

Random variable denoting the action in timestep t − 1 and its particular values, respectively

\(a^{\prime } ,~a_{t}\) :

Action in timestep t

\(G_{t}\) :

Random variable denoting the return in timestep t

\(\pi\) :

Policy

\(q^{\pi }\) :

Action value function (Q-function) related to the policy π

\(Q\left( {s^{\prime } ,a^{\prime } ;\theta ^{Q} } \right)\) :

Deep Q-network

\(\theta ^{Q}\) :

Deep Q-network parameters

\(Q_{{{\text{label}}}}\) :

Label for deep Q-network training

\(Q_{{{\text{target}}}} \left( {s^{\prime } ,a^{\prime } ;\theta ^{{Q_{{{\text{target}}}} }} } \right)\) :

Target deep Q-network

\(\theta ^{{Q_{{{\text{target}}}} }}\) :

Target deep Q-network parameters

\(L\) :

Squared error loss function

\(t\) :

Discrete time interval index (t = 1,2,…,T, where T is the total number of time intervals)

\(j,~k\) :

Bus (\(j,~k = 1,~2,~ \ldots ,~N_{N},\) where \(N_{N}\) is the total number of buses in a distribution network)

\(b\) :

Branch (\(b = 1,~2,~ \ldots ,~N_{{{\text{Br}}}},\) where \(N_{\text{Br}}\) is the total number of branches)

\({\text{sw}}\) :

Switch (\({\text{sw}} = 1,~2,~ \ldots ,~N_{{{\text{SW}}}},\) where \(N_{\text{SW}}\) is the number of switches in a network)

\({\text{episode}}\) :

Episode index

\({\mathcal{S}}\) :

Finite set of states

\({\mathcal{A}}\) :

Finite set of actions

\({\mathcal{R}}\) :

Finite set of immediate rewards

\(g,~b\) :

Element of nodal conductance and susceptance matrices, respectively

\(N_{{{\text{SW}}}}^{{{\text{max}}}}\) :

Maximum number of allowed switch operations during an optimization time interval

\(R,~X\) :

Branch resistance and reactance, respectively,

\(S_{b}^{{{\text{max}}}}\) :

Maximum allowed VA power flow in \(b{\text{th}} \) branch

\(U_{j}^{{{\text{min}}}} ,~U_{j}^{{{\text{max}}}}\) :

Minimum and maximum allowed kV voltage at \(j{\text{th}}\) bus

\(N\) :

Number of episodes

\(T\) :

Number of time intervals, i.e. number of timesteps per episode

CLoss :

Cost of energy losses, in $ per kWh

CSWs:

Cost of switching action for the \(s{\text{th}}\)switch, in $

\(C_{U}\) :

Penalty value if the bus voltage constraint is violated in any of the nodes

\(C_{S}\) :

Penalty value if the branch capacity constraint is violated for any of the branches

γ :

Discount factor

ε :

Exploration hyperparameter

\(\alpha\) :

Learning rate hyperparameter for Q-learning

DNR:

Distribution network reconfiguration

DDNR:

Dynamic distribution network reconfiguration

RL:

Reinforcement learning

MDP:

Markov decision process

DRL:

Deep reinforcement learning

DQN:

Deep Q-network

ReLU:

Rectified linear unit

References

  1. Samman MA, Mokhlis H, Mansor NN, Mohamad H, Suyono H, Sapari NM (2020) Fast optimal network reconfiguration with guided initialization based on a simplified network approach. IEEE Access 8:11948–11963

    Article  Google Scholar 

  2. Fathi V, Seyedi H, Ivatloo BM (2020) Reconfiguration of distribution systems in the presence of distributed generation considering protective constraints and uncertainties. Int Trans Electr Energy Syst 2020:1–25

    Google Scholar 

  3. Mishra S, Das D, Paul S (2017) A comprehensive review on power distribution network reconfiguration. Energy Syst 8:227–284

    Article  Google Scholar 

  4. Civanlar S, Grainger JJ, Yin H, Lee SSH (1988) Distribution feeder reconfiguration for loss reduction. IEEE Trans Power Deliv 3:1217–1223

    Article  Google Scholar 

  5. Shirmohamadi D, Hong HW (1989) Reconfiguration of electric distribution networks for resistive line loss reduction. IEEE Trans Power Deliv 2:1492–1498

    Article  Google Scholar 

  6. Baran M, Wu FF (1989) Network reconfiguration in distribution systems for loss reduction and load balancing. IEEE Trans Power Deliv 2:1401–1407

    Article  Google Scholar 

  7. Taleski R, Rajicic D (1997) Distribution network reconfiguration for energy loss reduction. IEEE Trans Power Syst 1:398–406

    Article  Google Scholar 

  8. Zhou Q, Shirmohammadi D, Liu WHE (1997) Distribution feeder reconfiguration for operation cost reduction. IEEE Trans Power Syst 2:730–735

    Article  Google Scholar 

  9. Borghetti A (2012) A mixed-integer linear programming approach for the computation of the minimum-losses radial configuration of electrical distribution networks. IEEE Trans Power Syst 3:1264–1273

    Article  Google Scholar 

  10. Ahmadi H, Marti JR (2015) Distribution system optimization based on a linear power flow formulation. IEEE Trans Power Deliv 1:25–33

    Article  Google Scholar 

  11. Lavorato M, Franko JF, Rider MJ, Romero R (2012) Imposing radiality constraints in distribution system optimization problems. IEEE Trans Power Syst 1:172–180

    Article  Google Scholar 

  12. Jabr RA, Singh R, Pal BC (2012) Minimum loss network reconfiguration using mixed integer convex programming. IEEE Trans Power Syst 2:1106–1115

    Article  Google Scholar 

  13. Haghighat H, Zeng B (2016) Distribution system reconfiguration under uncertain load and renewable generation. IEEE Trans Power Syst 4:2666–2675

    Article  Google Scholar 

  14. Gupta N, Swarnkar A, Niazi KR (2014) Distribution network reconfiguration for power quality and reliability improvement using genetic algorithms. Int J Electr Power Energy Syst 54:664–671

    Article  Google Scholar 

  15. Tsai M, Hsu F (2010) Application of grey correlation analysis in evolutionary programming for distribution system feeder reconfiguration. IEEE Trans Power Syst 2:1126–1133

    Article  Google Scholar 

  16. Wu W, Tsai M (2011) Application of enhanced integer coded particle swarm optimization for distribution system feeder reconfiguration. IEEE Trans Power Syst 3:1591–1599

    Article  Google Scholar 

  17. Golshannavaz S, Afsharnia S, Aminifar F (2014) Smart distribution grid: optimal dayahead scheduling with reconfigurable topology. IEEE Transactions Smart Grid 5:2402–2411

    Article  Google Scholar 

  18. Li Z, Chen X, Yu K, Zhao B, Liu H (2008) A novel approach for dynamic reconfiguration of the distribution network via multi-agent system. In: Third International conference on electric utility deregulation and restructuring and power technologies, pp 1305–1311

  19. Ramos ER, Exposito AG, Santos JR, Iborra FL (2005) Path-based distribution network modeling: application to reconfiguration for loss reduction. IEEE Trans Power Syst 2:556–564

    Article  Google Scholar 

  20. Broadwater RP, Khan AH, Shaalan HE, Lee RE (1993) Time varying load analysis to reduce distribution losses through reconfiguration. IEEE Trans Power Syst 1:294–300

    Article  Google Scholar 

  21. Chen CS, Cho MY (1993) Energy loss reduction by critical switches. IEEE Trans Power Delivery 3:1246–1253

    Article  Google Scholar 

  22. López E, Opazo H, García L, Bastard P (2004) On line reconfiguration considering variability demand: applications to real networks. IEEE Trans Power Syst 1:549–553

    Article  Google Scholar 

  23. Shariatkhah MH, Haghifam MR, Salehi J, Moser A (2012) Duration based reconfiguration of electric distribution networks using dynamic programming and harmony search algorithm. Int J Electr Power Energy Syst 1:1–10

    Article  Google Scholar 

  24. Zidan A, El-Saadany EF (2013) Distribution system reconfiguration for energy loss reduction considering the variability of load and local renewable generation. Energy 59:698–707

    Article  Google Scholar 

  25. Milani AE, Haghifam MR (2013) An evolutionary approach for optimal time interval determination in distribution network reconfiguration under variable load. Math Comput Model 1–2:68–77

    Article  MathSciNet  Google Scholar 

  26. Mazza A, Chicco G, Andrei H, Rubino M (2015) Determination of the relevant periods for intraday distribution system minimum loss reconfiguration. Int Trans Electr Energy Syst 10:1992–2023

    Article  Google Scholar 

  27. Karimianfard H, Haghighat H (2019) An initial-point strategy for optimizing distribution system reconfiguration. Electric Power Syst Res 176:105943–105950

    Article  Google Scholar 

  28. Jafari A, Ganjehlou HG, Darbandi FB, Mohammdi-Ivatloo B, Abapour M (2020) Dynamic and multi-objective reconfiguration of distribution network using a novel hybrid algorithm with parallel processing capability. Appl Soft Comput J 90:106146–106165

    Article  Google Scholar 

  29. Kovački NV, Vidović PM, Sarić AT (2018) Scalable algorithm for the dynamic reconfiguration of the distribution network using the Lagrange relaxation approach. Int J Electr Power Energy Syst 94:188–202

    Article  Google Scholar 

  30. Gao Y, Shi J, Wang W, Yu N (2019) Dynamic distribution network reconfiguration using reinforcement learning. In: IEEE international conference on communications, control, and computing technologies for smart grids (SmartGridComm), pp 1–7

  31. Liu Z, Liu Y, Qu G, Wang X, Wang X (2019) Intra-day dynamic network reconfiguration based on probability analysis considering the deployment of remote control switches. IEEE Access 7:145272–145281

    Article  Google Scholar 

  32. Ahmadi I, Ahmadigorji M, Tohidifar E (2018) A novel approach for power loss reduction in distribution networks considering budget constraint. Int Trans Electr Energy Syst 12:1–27

    Google Scholar 

  33. Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT Press, Cambridge

    MATH  Google Scholar 

  34. Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 3–4:279–292

    MATH  Google Scholar 

  35. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 7540:529–533

    Article  Google Scholar 

  36. Dugan R, McDermott T (2011) An open source platform for collaborating on smart grid research. Power and energy society general meeting 2011 IEEE, pp 1–7

  37. Shih-An Y, Chan-Nan L (2009) Distribution feeder scheduling considering variable load profile and outage costs. IEEE Trans Power Syst 2:652–660

    Article  Google Scholar 

  38. Kashem MA, Ganapathy V, Jasmon GB, Buhari MI (2000) A novel method for loss minimization in distribution networks. In: International conference on electric utility deregulation and restructuring and power technologies. Proceedings, pp251–256

Download references

Author information

Authors and Affiliations

  1. Faculty of Technical Sciences, University of Novi Sad, Novi Sad, Serbia

    Ognjen B. Kundačina & Predrag M. Vidović

  2. Schneider Electric DMS NS, Novi Sad, Serbia

    Milan R. Petković

Authors
  1. Ognjen B. Kundačina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Predrag M. Vidović
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Milan R. Petković
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Predrag M. Vidović.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kundačina, O.B., Vidović, P.M. & Petković, M.R. Solving dynamic distribution network reconfiguration using deep reinforcement learning. Electr Eng 104, 1487–1501 (2022). https://doi.org/10.1007/s00202-021-01399-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00202-021-01399-y

Keywords

Search

Navigation