Abstract
In this chapter, we move to address the fundamental optimization problems in smart grid from a model-free perspective due to the randomness of the large-scale renewable energies and the flexibility of the load units. That is to say, these fundamental problems cannot be directly modeled as deterministic optimization problems. So we design learning-based approaches to address the uncertainty and randomness.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Row-stochastic communication matrix means each agent computes a weighted sum of received data with some weighting factors. The column-stochastic requires these factors to be compatible in sense that each column of communication matrix sums to one.
- 2.
Since the energy demand and consumption of loads is affected by many factors, the state transition is rather difficult to obtain. Therefore, we next employ a model-free Q-learning method to solve the dynamic retail pricing problem.
- 3.
Thus the EDP considered in these situations falls under the umbrella of derivative-free blackbox optimization. It is commonly recognized that if the (generalized) gradient information is reliable and obtainable at a reasonable cost, derivative-free blackbox optimization almost never outperform modern gradient-based methods [76].
References
Zaman, M., Elsayed, S.M., Ray, T., Sarker, R.A.: Evolutionary algorithms for dynamic economic dispatch problems. IEEE Trans. Power Syst. 31(2), 1486–1495 (2015)
Jeyakumar, D., Jayabarathi, T., Raghunathan, T.: Particle swarm optimization for various types of economic dispatch problems. Int. J. Electr. Power Energy Syst. 28(1), 36–42 (2006)
Walters, D.C., Sheble, G.B.: Genetic algorithm solution of economic dispatch with valve point loading. IEEE Trans. Power Syst. 8(3), 1325–1332 (1993)
Adarsh, B., Raghunathan, T., Jayabarathi, T., Yang, X.-S.: Economic dispatch using chaotic bat algorithm. Energy 96, 666–675 (2016)
Li, F., Qin, J., Kang, Y., Zheng, W.X.: Consensus based distributed reinforcement learning for nonconvex economic power dispatch in microgrids. In: International Conference on Neural Information Processing, pp. 831–839. Springer, Berlin (2017)
Xie, J., Yu, Q., Cao, C.: A distributed randomized gradient-free algorithm for the non-convex economic dispatch problem. Energies 11(1), 244 (2018)
Qin, J., Ma, Q., Shi, Y., Wang, L.: Recent advances in consensus of multi-agent systems: a brief survey. IEEE Trans. Ind. Electron. 64(6), 4972–4983 (2017)
Oliva, G., Setola, R., Hadjicostis, C.N.: Distributed finite-time average-consensus with limited computational and storage capability. IEEE Trans. Control Netw. Syst. 4(2), 380–391 (2016)
Alba, E., Luque, G., Nesmachnow, S.: Parallel metaheuristics: recent advances and new trends. Intl. Trans. Oper. Res. 20(1), 1–48 (2013)
Elsayed, W.T., El-Saadany, E.F.: A fully decentralized approach for solving the economic dispatch problem. IEEE Trans. Power Syst. 30(4), 2179–2189 (2014)
Chen, G., Ren, J., Feng, E.N.: Distributed finite-time economic dispatch of a network of energy resources. IEEE Trans. Smart Grid 8(2), 822–832 (2017)
Yang, S., Tan, S., Xu, J.X.: Consensus based approach for economic dispatch problem in a smart grid. IEEE Trans. Power Syst. 28(4), 4416–4426 (2013)
Elsayed, W.T., Hegazy, Y.G., El-bages, M.S., Bendary, F.M.: Improved random drift particle swarm optimization with self-adaptive mechanism for solving the power economic dispatch problem. IEEE Trans. Ind. Inform. 13(3), 1017–1026 (2017)
Liang, H., Liu, Y., Shen, Y., Li, F., Man, Y.: A hybrid bat algorithm for economic dispatch with random wind power. IEEE Trans. Power Syst. 33(5), 5052–5061 (2018)
He, X., Rao, Y., Huang, J.: A novel algorithm for economic load dispatch of power systems. Neurocomputing 171, 1454–1461 (2016)
Sinha, N., Chakrabarti, R., Chattopadhyay, P.K.: Evolutionary programming techniques for economic load dispatch. IEEE Trans. Evol. Comput. 7(1), 83–94 (2003)
Wood, A.J., Wollenberg, B.F., Sheblé, G.B.: Power Generation, Operation, and Control. Wiley, New York (2013)
Zarco, P., Exposito, A.G.: Power system parameter estimation: a survey. IEEE Trans. Power Syst. 15(1), 216–222 (2000)
Zhang, L., Abur, A.: Identifying parameter errors via multiple measurement scans. IEEE Trans. Power Syst. 28(4), 3916–3923 (2013)
Bertsekas, D.: Convex Optimization Theory, vol. 1. Athena Scientific, Nashua (2009)
Audet, C., Hare, W.: Derivative-Free and Blackbox Optimization, vol. 2. Springer, Berlin (2017)
Sundaram, S., Hadjicostis, C.N.: Finite-time distributed consensus in graphs with time-invariant topologies. In: 2007 American Control Conference, pp. 711–716. IEEE (2007)
Molzahn, D.K., Dörfler, F., Sandberg, H., Low, S.H., Chakrabarti, S., Baldick, R., Lavaei, J.: A survey of distributed optimization and control algorithms for electric power systems. IEEE Trans. Smart Grid 8(6), 2941–2962 (2017)
Deng, R., Xiao, G., Lu, R., Chen, J.: Fast distributed demand response with spatially and temporally coupled constraints in smart grid. IEEE Trans. Ind. Inf. 11(6), 1597–1606 (2015)
Ding, L., Yin, G.Y., Zheng, W.X., Han, Q.-L., et al.: Distributed energy management for smart grids with an event-triggered communication scheme. IEEE Trans. Control Syst. Technol. 27(5), 1950–1961 (2018)
Hao, H., Corbin, C.D., Kalsi, K., Pratt, R.G.: Transactive control of commercial buildings for demand response. IEEE Trans. Power Syst. 32(1), 774–783 (2016)
Shen, B., Ghatikar, G., Lei, Z., Li, J., Wikler, G., Martin, P.: The role of regulatory reforms, market changes, and technology development to make demand response a viable resource in meeting energy challenges. Appl. Energy 130, 814–823 (2014)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Vandael, S., Claessens, B., Ernst, D., Holvoet, T., Deconinck, G.: Reinforcement learning of heuristic EV fleet charging in a day-ahead electricity market. IEEE Trans. Smart Grid 6(4), 1795–1805 (2015)
Kim, B.-G., Zhang, Y., Van Der Schaar, M., Lee, J.-W.: Dynamic pricing and energy consumption scheduling with reinforcement learning. IEEE Trans. Smart Grid 7(5), 2187–2198 (2015)
Foruzan, E., Soh, L.-K., Asgarpoor, S.: Reinforcement learning approach for optimal distributed energy management in a microgrid. IEEE Trans. Power Syst. 33(5), 5749–5758 (2018)
Imani, M., Ghoreishi, S.F.: Scalable inverse reinforcement learning through multifidelity Bayesian optimization. IEEE Trans. Neural Netw. Learn. Syst. 33(8), 4125–4132 (2022)
Sadeghi, M., Erol-Kantarci, M.: Power loss minimization in microgrids using bayesian reinforcement learning with coalition formation. In: 2019 IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), pp. 1–6. IEEE (2019)
Liu, T., Tian, B., Ai, Y., Wang, F.-Y.: Parallel reinforcement learning-based energy efficiency improvement for a cyber-physical system. IEEE/CAA J. Autom. Sin. 7(2), 617–626 (2020)
Ahrarinouri, M., Rastegar, M., Seifi, A.R.: Multiagent reinforcement learning for energy management in residential buildings. IEEE Trans. Ind. Inf. 17(1), 659–666 (2020)
Zhang, X., Biagioni, D., Cai, M., Graf, P., Rahman, S.: An edge-cloud integrated solution for buildings demand response using reinforcement learning. IEEE Tran. Smart Grid 12(1), 420–431 (2020)
Zhao, Y., Cai, Y., Song, Q.: Energy control of plug-in hybrid electric vehicles using model predictive control with route preview. IEEE/CAA J. Autom. Sin. 8(12), 1948–4854 (2018)
Deng, R., Yang, Z., Hou, F., Chow, M.Y., Chen, J.: Distributed real-time demand response in multiseller-multibuyer smart distribution grid. IEEE Trans. Power Syst. 30(5), 2364–2374 (2015)
Hao, H., Wu, D., Lian, J., Yang, T.: Optimal coordination of building loads and energy storage for power grid and end user services. IEEE Trans. Smart Grid 9(5), 4335–4345 (2017)
Vanthournout, K., Dupont, B., Foubert, W., Stuckens, C., Claessens, S.: An automated residential demand response pilot experiment, based on day-ahead dynamic pricing. Appl. Energy 155, 195–203 (2015)
Mohsenian-Rad, A.H., Wong, V.W.S., Jatskevich, J., Schober, R., Leon-Garcia, A.: Autonomous demand-side management based on game-theoretic energy consumption scheduling for the future smart grid. IEEE Trans. Smart Grid 1(3), 320–331 (2010)
Dong, Q., Yu, L., Song, W., Yang, J., Wu, Y., Qi, J.: Fast distributed demand response algorithm in smart grid. IEEE/CAA J. Autom. Sin. 4(2), 280–296 (2017)
Jin, M., Feng, W., Marnay, C., Spanos, C.: Microgrid to enable optimal distributed energy retail and end-user demand response. Appl. Energy 210, 1321–1335 (2018)
Lu, R., Hong, S.H., Zhang, X.: A dynamic pricing demand response algorithm for smart grid: reinforcement learning approach. Appl. Energy 220, 220–230 (2018)
Yu, M., Hong, S.H., Ding, Y., Ye, X.: An incentive-based demand response (DR) model considering composited DR resources. IEEE Trans. Ind. Electron. 66(2), 1488–1498 (2018)
Wan, Y., Qin, J., Li, F., Yu, X., Kang, Y.: Game theoretic-based distributed charging strategy for PEVs in a smart charging station. IEEE Trans. Smart Grid 12(1), 538–547 (2020)
Yu, L., Xie, W., Xie, D., Zou, Y., Zhang, D., Sun, Z., Zhang, L., Zhang, Y., Jiang, T.: Deep reinforcement learning for smart home energy management. IEEE Internet Things J. 7(4), 2751–2762 (2019)
Yu, L., Sun, Y., Xu, Z., Shen, C., Yue, D., Jiang, T., Guan, X.: Multi-agent deep reinforcement learning for HVAC control in commercial buildings. IEEE Trans. Smart Grid 12(1), 407–419 (2020)
Jahangir, H., Gougheri, S.S., Vatandoust, B., Golkar, M.A., Ahmadian, A., Hajizadeh, A.: Plug-in electric vehicle behavior modeling in energy market: a novel deep learning-based approach with clustering technique. IEEE Trans. Smart Grid 11(6), 4738–4748 (2020)
Li, F., Qin, J., Zheng, W.X.: Distributed q-learning-based online optimization algorithm for unit commitment and dispatch in smart grid. IEEE Trans. Cybern. 50(9), 4146–4156 (2019)
Du, Y., Li, F.: Intelligent multi-microgrid energy management based on deep neural network and model-free reinforcement learning. IEEE Trans. Smart Grid 11(2), 1066–1076 (2019)
Sdge.com. Home—San Diego Gas & Electric. [Online]. Available: https://www.sdge.com
Real-Time Hourly Prices. Commonwealth Edison Company. [Online]. Available: https://rrtp.comed.com/live-prices/
Koenig, S., Simmons, R.G.: The effect of representation and knowledge on goal-directed exploration with reinforcement-learning algorithms. Mach. Learn. 22(1), 227–250 (1996)
Yu, X., Xue, Y.: Smart grids: a cyber-physical systems perspective. Proc. IEEE 104(5), 1058–1070 (2016)
Boy, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)
Vaccaro, A., Loia, V., Formato, G., Wall, P., Terzija, V.: A self-organizing architecture for decentralized smart microgrids synchronization, control, and monitoring. IEEE Trans. Ind. Inf. 11(1), 289–298 (2014)
Zhang, Z., Ying, X., Chow, M.-Y.: Decentralizing the economic dispatch problem using a two-level incremental cost consensus algorithm in a smart grid environment. In: 2011 North American Power Symposium, pp. 1–7. IEEE (2011)
Zhang, Z., Chow, M.-Y.: Convergence analysis of the incremental cost consensus algorithm under different communication network topologies in a smart grid. IEEE Trans. Power Syst. 27(4), 1761–1768 (2012)
Kar, S., Hug, G.: Distributed robust economic dispatch in power systems: a consensus + innovations approach. In: Proceedings of IEEE Power Energy Society General Meeting (PESGM), pp. 1–8 (2012)
Hug, G., Kar, S., Wu, C.: Consensus + innovations approach for distributed multiagent coordination in a microgrid. IEEE Trans. Smart Grid 6(4), 1893–1903 (2015)
Loia, V., Vaccaro, A.: Decentralized economic dispatch in smart grids by self-organizing dynamic agents. IEEE Trans. Syst. Man Cybern. Syst. 44(4), 397–408 (2014)
Qin, J., Fu, W., Zheng, W.X., Gao, H.: On the bipartite consensus for generic linear multiagent systems with input saturation. IEEE Trans. Cybern. 47(8), 1948–1958 (2016)
Qin, J., Ma, Q., Gao, H., Shi, Y., Kang, Y.: On group synchronization for interacting clusters of heterogeneous systems. IEEE Trans. Cybern. 47(12), 4122–4133 (2016)
Qin, J., Fu, W., Gao, H., Zheng, W.X.: Distributed k-means algorithm and fuzzy c-means algorithm for sensor networks based on multiagent consensus theory. IEEE Trans. Cybern. 47(3), 772–783 (2016)
Wen, G., Yu, W., Yu, X., Cao, J.: Designing adaptive consensus-based scheme for economic dispatch of smart grid. In: 2016 Eighth International Conference on Advanced Computational Intelligence (ICACI), pp. 236–241. IEEE (2016)
Guo, F., Wen, C., Mao, J., Song, Y.-D.: Distributed economic dispatch for smart grids with random wind power. IEEE Trans. Smart Grid 7(3), 1572–1583 (2015)
Xing, H., Mou, Y., Fu, M., Lin, Z.: Distributed bisection method for economic power dispatch in smart grid. IEEE Trans. Power Syst. 30(6), 3024–3035 (2014)
Yang, T., Wu, D., Sun, Y., Lian, J.: Minimum-time consensus-based approach for power system applications. IEEE Trans. Ind. Electron. 63(2), 1318–1328 (2015)
Wen, G., Yu, X., Liu, Z.-W., Yu, W.: Adaptive consensus-based robust strategy for economic dispatch of smart grids subject to communication uncertainties. IEEE Trans. Ind. Inf. 14(6), 2484–2496 (2017)
Wen, G., Hu, G., Hu, J., Shi, X., Chen, G.: Frequency regulation of source-grid-load systems: a compound control strategy. IEEE Trans. Ind. Inf. 12(1), 69–78 (2015)
Li, C., Yu, X., Huang, T., He, X.: Distributed optimal consensus over resource allocation network and its application to dynamical economic dispatch. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2407–2418 (2017)
Qin, J., Wan, Y., Yu, X., Li, F., Li, C.: Consensus-based distributed coordination between economic dispatch and demand response. IEEE Trans. Smart Grid 10(4), 3709–3719 (2018)
Li, F., Qin, J., Kang, Y.: Multi-agent system based distributed pattern search algorithm for non-convex economic load dispatch in smart grid. IEEE Trans. Power Syst. 34(3), 2093–2102 (2018)
Zhao, C., He, J., Cheng, P., Chen, J.: Analysis of consensus-based distributed economic dispatch under stealthy attacks. IEEE Trans. Ind. Electron. 64(6), 5107–5117 (2016)
Audet, C., Hare, W.: Derivative-Free and Blackbox Optimization, vol. 2. Springer, Berlin (2017)
Durai, S., Subramanian, S., Ganesan, S.: Improved parameters for economic dispatch problems by teaching learning optimization. Int. J. Electr. Power Energy Syst. 67, 11–24 (2015)
Hung, S.-M., Givigi, S.N.: A q-learning approach to flocking with UAVs in a stochastic environment. IEEE Trans. Cybern. 47(1), 186–197 (2016)
Zhang, H., Jiang, H., Luo, Y., Xiao, G.: Data-driven optimal consensus control for discrete-time multi-agent systems with unknown dynamics using reinforcement learning method. IEEE Trans. Ind. Electron. 64(5), 4091–4100 (2016)
Zhang, H., Jiang, H., Luo, C., Xiao, G.: Discrete-time nonzero-sum games for multiplayer using policy-iteration-based adaptive dynamic programming algorithms. IEEE Trans. Cybern. 47(10), 3331–3340 (2016)
Olfati-Saber, R., Murray, R.M.: Consensus problems in networks of agents with switching topology and time-delays. IEEE Trans. Automat. Contr. 49(9), 1520–1533 (2004)
Saravanan, B., Vasudevan, E., Kothari, D.: Unit commitment problem solution using invasive weed optimization algorithm. Int. J. Electr. Power Energy Syst. 55, 21–28 (2014)
Lujano-Rojas, J., Osório, G., Catalão, J.: New probabilistic method for solving economic dispatch and unit commitment problems incorporating uncertainty due to renewable energy integration. Int. J. Electr. Power Energy Syst. 78, 61–71 (2016)
Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)
Wang, C., Li, X., Wang, Z., Dong, X., Liang, Z., Liu, X., Liang, J., Han, X.: Day-ahead unit commitment method considering time sequence feature of wind power forecast error. Int. J. Electr. Power Energy Syst. 98, 156–166 (2018)
Carrión, M., Arroyo, J.M.: A computationally efficient mixed-integer linear formulation for the thermal unit commitment problem. IEEE Trans. Power Syst. 21(3), 1371–1378 (2006)
Wang, Y., Zhao, S., Zhou, Z., Botterud, A., Xu, Y., Chen, R.: Risk adjustable day-ahead unit commitment with wind power based on chance constrained goal programming. IEEE Trans. Sustain. Energy 8(2), 530–541 (2016)
Bellifemine, F.L., Caire, G., Greenwood, D.: Developing Multi-agent Systems with JADE. Wiley, New York (2007)
Xu, Y., Zhang, W., Liu, W.: Distributed dynamic programming-based approach for economic dispatch in smart grids. IEEE Trans. Ind. Inf. 11(1), 166–175 (2014)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Qin, J., Wan, Y., Li, F., Kang, Y., Fu, W. (2023). Model-Free Distributed Optimization. In: Distributed Economic Operation in Smart Grid: Model-Based and Model-Free Perspectives. Studies in Systems, Decision and Control, vol 455. Springer, Singapore. https://doi.org/10.1007/978-981-19-8594-2_4
Download citation
DOI: https://doi.org/10.1007/978-981-19-8594-2_4
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-8593-5
Online ISBN: 978-981-19-8594-2
eBook Packages: EngineeringEngineering (R0)