Skip to main content

Part of the book series: Studies in Systems, Decision and Control ((SSDC,volume 455))

  • 175 Accesses

Abstract

In this chapter, we move to address the fundamental optimization problems in smart grid from a model-free perspective due to the randomness of the large-scale renewable energies and the flexibility of the load units. That is to say, these fundamental problems cannot be directly modeled as deterministic optimization problems. So we design learning-based approaches to address the uncertainty and randomness.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Row-stochastic communication matrix means each agent computes a weighted sum of received data with some weighting factors. The column-stochastic requires these factors to be compatible in sense that each column of communication matrix sums to one.

  2. 2.

    Since the energy demand and consumption of loads is affected by many factors, the state transition is rather difficult to obtain. Therefore, we next employ a model-free Q-learning method to solve the dynamic retail pricing problem.

  3. 3.

    Thus the EDP considered in these situations falls under the umbrella of derivative-free blackbox optimization. It is commonly recognized that if the (generalized) gradient information is reliable and obtainable at a reasonable cost, derivative-free blackbox optimization almost never outperform modern gradient-based methods [76].

References

  1. Zaman, M., Elsayed, S.M., Ray, T., Sarker, R.A.: Evolutionary algorithms for dynamic economic dispatch problems. IEEE Trans. Power Syst. 31(2), 1486–1495 (2015)

    Article  Google Scholar 

  2. Jeyakumar, D., Jayabarathi, T., Raghunathan, T.: Particle swarm optimization for various types of economic dispatch problems. Int. J. Electr. Power Energy Syst. 28(1), 36–42 (2006)

    Article  Google Scholar 

  3. Walters, D.C., Sheble, G.B.: Genetic algorithm solution of economic dispatch with valve point loading. IEEE Trans. Power Syst. 8(3), 1325–1332 (1993)

    Article  Google Scholar 

  4. Adarsh, B., Raghunathan, T., Jayabarathi, T., Yang, X.-S.: Economic dispatch using chaotic bat algorithm. Energy 96, 666–675 (2016)

    Article  Google Scholar 

  5. Li, F., Qin, J., Kang, Y., Zheng, W.X.: Consensus based distributed reinforcement learning for nonconvex economic power dispatch in microgrids. In: International Conference on Neural Information Processing, pp. 831–839. Springer, Berlin (2017)

    Google Scholar 

  6. Xie, J., Yu, Q., Cao, C.: A distributed randomized gradient-free algorithm for the non-convex economic dispatch problem. Energies 11(1), 244 (2018)

    Article  Google Scholar 

  7. Qin, J., Ma, Q., Shi, Y., Wang, L.: Recent advances in consensus of multi-agent systems: a brief survey. IEEE Trans. Ind. Electron. 64(6), 4972–4983 (2017)

    Article  Google Scholar 

  8. Oliva, G., Setola, R., Hadjicostis, C.N.: Distributed finite-time average-consensus with limited computational and storage capability. IEEE Trans. Control Netw. Syst. 4(2), 380–391 (2016)

    Article  MATH  Google Scholar 

  9. Alba, E., Luque, G., Nesmachnow, S.: Parallel metaheuristics: recent advances and new trends. Intl. Trans. Oper. Res. 20(1), 1–48 (2013)

    Article  MATH  Google Scholar 

  10. Elsayed, W.T., El-Saadany, E.F.: A fully decentralized approach for solving the economic dispatch problem. IEEE Trans. Power Syst. 30(4), 2179–2189 (2014)

    Article  Google Scholar 

  11. Chen, G., Ren, J., Feng, E.N.: Distributed finite-time economic dispatch of a network of energy resources. IEEE Trans. Smart Grid 8(2), 822–832 (2017)

    Google Scholar 

  12. Yang, S., Tan, S., Xu, J.X.: Consensus based approach for economic dispatch problem in a smart grid. IEEE Trans. Power Syst. 28(4), 4416–4426 (2013)

    Article  Google Scholar 

  13. Elsayed, W.T., Hegazy, Y.G., El-bages, M.S., Bendary, F.M.: Improved random drift particle swarm optimization with self-adaptive mechanism for solving the power economic dispatch problem. IEEE Trans. Ind. Inform. 13(3), 1017–1026 (2017)

    Article  Google Scholar 

  14. Liang, H., Liu, Y., Shen, Y., Li, F., Man, Y.: A hybrid bat algorithm for economic dispatch with random wind power. IEEE Trans. Power Syst. 33(5), 5052–5061 (2018)

    Article  Google Scholar 

  15. He, X., Rao, Y., Huang, J.: A novel algorithm for economic load dispatch of power systems. Neurocomputing 171, 1454–1461 (2016)

    Article  Google Scholar 

  16. Sinha, N., Chakrabarti, R., Chattopadhyay, P.K.: Evolutionary programming techniques for economic load dispatch. IEEE Trans. Evol. Comput. 7(1), 83–94 (2003)

    Article  Google Scholar 

  17. Wood, A.J., Wollenberg, B.F., Sheblé, G.B.: Power Generation, Operation, and Control. Wiley, New York (2013)

    Google Scholar 

  18. Zarco, P., Exposito, A.G.: Power system parameter estimation: a survey. IEEE Trans. Power Syst. 15(1), 216–222 (2000)

    Article  Google Scholar 

  19. Zhang, L., Abur, A.: Identifying parameter errors via multiple measurement scans. IEEE Trans. Power Syst. 28(4), 3916–3923 (2013)

    Article  Google Scholar 

  20. Bertsekas, D.: Convex Optimization Theory, vol. 1. Athena Scientific, Nashua (2009)

    Google Scholar 

  21. Audet, C., Hare, W.: Derivative-Free and Blackbox Optimization, vol. 2. Springer, Berlin (2017)

    Google Scholar 

  22. Sundaram, S., Hadjicostis, C.N.: Finite-time distributed consensus in graphs with time-invariant topologies. In: 2007 American Control Conference, pp. 711–716. IEEE (2007)

    Google Scholar 

  23. Molzahn, D.K., Dörfler, F., Sandberg, H., Low, S.H., Chakrabarti, S., Baldick, R., Lavaei, J.: A survey of distributed optimization and control algorithms for electric power systems. IEEE Trans. Smart Grid 8(6), 2941–2962 (2017)

    Article  Google Scholar 

  24. Deng, R., Xiao, G., Lu, R., Chen, J.: Fast distributed demand response with spatially and temporally coupled constraints in smart grid. IEEE Trans. Ind. Inf. 11(6), 1597–1606 (2015)

    Article  Google Scholar 

  25. Ding, L., Yin, G.Y., Zheng, W.X., Han, Q.-L., et al.: Distributed energy management for smart grids with an event-triggered communication scheme. IEEE Trans. Control Syst. Technol. 27(5), 1950–1961 (2018)

    Article  Google Scholar 

  26. Hao, H., Corbin, C.D., Kalsi, K., Pratt, R.G.: Transactive control of commercial buildings for demand response. IEEE Trans. Power Syst. 32(1), 774–783 (2016)

    Article  Google Scholar 

  27. Shen, B., Ghatikar, G., Lei, Z., Li, J., Wikler, G., Martin, P.: The role of regulatory reforms, market changes, and technology development to make demand response a viable resource in meeting energy challenges. Appl. Energy 130, 814–823 (2014)

    Article  Google Scholar 

  28. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)

    Google Scholar 

  29. Vandael, S., Claessens, B., Ernst, D., Holvoet, T., Deconinck, G.: Reinforcement learning of heuristic EV fleet charging in a day-ahead electricity market. IEEE Trans. Smart Grid 6(4), 1795–1805 (2015)

    Article  Google Scholar 

  30. Kim, B.-G., Zhang, Y., Van Der Schaar, M., Lee, J.-W.: Dynamic pricing and energy consumption scheduling with reinforcement learning. IEEE Trans. Smart Grid 7(5), 2187–2198 (2015)

    Article  Google Scholar 

  31. Foruzan, E., Soh, L.-K., Asgarpoor, S.: Reinforcement learning approach for optimal distributed energy management in a microgrid. IEEE Trans. Power Syst. 33(5), 5749–5758 (2018)

    Article  Google Scholar 

  32. Imani, M., Ghoreishi, S.F.: Scalable inverse reinforcement learning through multifidelity Bayesian optimization. IEEE Trans. Neural Netw. Learn. Syst. 33(8), 4125–4132 (2022)

    Article  Google Scholar 

  33. Sadeghi, M., Erol-Kantarci, M.: Power loss minimization in microgrids using bayesian reinforcement learning with coalition formation. In: 2019 IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), pp. 1–6. IEEE (2019)

    Google Scholar 

  34. Liu, T., Tian, B., Ai, Y., Wang, F.-Y.: Parallel reinforcement learning-based energy efficiency improvement for a cyber-physical system. IEEE/CAA J. Autom. Sin. 7(2), 617–626 (2020)

    Article  Google Scholar 

  35. Ahrarinouri, M., Rastegar, M., Seifi, A.R.: Multiagent reinforcement learning for energy management in residential buildings. IEEE Trans. Ind. Inf. 17(1), 659–666 (2020)

    Article  Google Scholar 

  36. Zhang, X., Biagioni, D., Cai, M., Graf, P., Rahman, S.: An edge-cloud integrated solution for buildings demand response using reinforcement learning. IEEE Tran. Smart Grid 12(1), 420–431 (2020)

    Article  Google Scholar 

  37. Zhao, Y., Cai, Y., Song, Q.: Energy control of plug-in hybrid electric vehicles using model predictive control with route preview. IEEE/CAA J. Autom. Sin. 8(12), 1948–4854 (2018)

    Article  Google Scholar 

  38. Deng, R., Yang, Z., Hou, F., Chow, M.Y., Chen, J.: Distributed real-time demand response in multiseller-multibuyer smart distribution grid. IEEE Trans. Power Syst. 30(5), 2364–2374 (2015)

    Article  Google Scholar 

  39. Hao, H., Wu, D., Lian, J., Yang, T.: Optimal coordination of building loads and energy storage for power grid and end user services. IEEE Trans. Smart Grid 9(5), 4335–4345 (2017)

    Article  Google Scholar 

  40. Vanthournout, K., Dupont, B., Foubert, W., Stuckens, C., Claessens, S.: An automated residential demand response pilot experiment, based on day-ahead dynamic pricing. Appl. Energy 155, 195–203 (2015)

    Article  Google Scholar 

  41. Mohsenian-Rad, A.H., Wong, V.W.S., Jatskevich, J., Schober, R., Leon-Garcia, A.: Autonomous demand-side management based on game-theoretic energy consumption scheduling for the future smart grid. IEEE Trans. Smart Grid 1(3), 320–331 (2010)

    Article  Google Scholar 

  42. Dong, Q., Yu, L., Song, W., Yang, J., Wu, Y., Qi, J.: Fast distributed demand response algorithm in smart grid. IEEE/CAA J. Autom. Sin. 4(2), 280–296 (2017)

    Article  Google Scholar 

  43. Jin, M., Feng, W., Marnay, C., Spanos, C.: Microgrid to enable optimal distributed energy retail and end-user demand response. Appl. Energy 210, 1321–1335 (2018)

    Article  Google Scholar 

  44. Lu, R., Hong, S.H., Zhang, X.: A dynamic pricing demand response algorithm for smart grid: reinforcement learning approach. Appl. Energy 220, 220–230 (2018)

    Article  Google Scholar 

  45. Yu, M., Hong, S.H., Ding, Y., Ye, X.: An incentive-based demand response (DR) model considering composited DR resources. IEEE Trans. Ind. Electron. 66(2), 1488–1498 (2018)

    Article  Google Scholar 

  46. Wan, Y., Qin, J., Li, F., Yu, X., Kang, Y.: Game theoretic-based distributed charging strategy for PEVs in a smart charging station. IEEE Trans. Smart Grid 12(1), 538–547 (2020)

    Article  Google Scholar 

  47. Yu, L., Xie, W., Xie, D., Zou, Y., Zhang, D., Sun, Z., Zhang, L., Zhang, Y., Jiang, T.: Deep reinforcement learning for smart home energy management. IEEE Internet Things J. 7(4), 2751–2762 (2019)

    Article  Google Scholar 

  48. Yu, L., Sun, Y., Xu, Z., Shen, C., Yue, D., Jiang, T., Guan, X.: Multi-agent deep reinforcement learning for HVAC control in commercial buildings. IEEE Trans. Smart Grid 12(1), 407–419 (2020)

    Article  Google Scholar 

  49. Jahangir, H., Gougheri, S.S., Vatandoust, B., Golkar, M.A., Ahmadian, A., Hajizadeh, A.: Plug-in electric vehicle behavior modeling in energy market: a novel deep learning-based approach with clustering technique. IEEE Trans. Smart Grid 11(6), 4738–4748 (2020)

    Article  Google Scholar 

  50. Li, F., Qin, J., Zheng, W.X.: Distributed q-learning-based online optimization algorithm for unit commitment and dispatch in smart grid. IEEE Trans. Cybern. 50(9), 4146–4156 (2019)

    Article  Google Scholar 

  51. Du, Y., Li, F.: Intelligent multi-microgrid energy management based on deep neural network and model-free reinforcement learning. IEEE Trans. Smart Grid 11(2), 1066–1076 (2019)

    Article  Google Scholar 

  52. Sdge.com. Home—San Diego Gas & Electric. [Online]. Available: https://www.sdge.com

  53. Real-Time Hourly Prices. Commonwealth Edison Company. [Online]. Available: https://rrtp.comed.com/live-prices/

  54. Koenig, S., Simmons, R.G.: The effect of representation and knowledge on goal-directed exploration with reinforcement-learning algorithms. Mach. Learn. 22(1), 227–250 (1996)

    Article  MATH  Google Scholar 

  55. Yu, X., Xue, Y.: Smart grids: a cyber-physical systems perspective. Proc. IEEE 104(5), 1058–1070 (2016)

    Article  Google Scholar 

  56. Boy, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)

    Google Scholar 

  57. Vaccaro, A., Loia, V., Formato, G., Wall, P., Terzija, V.: A self-organizing architecture for decentralized smart microgrids synchronization, control, and monitoring. IEEE Trans. Ind. Inf. 11(1), 289–298 (2014)

    Article  Google Scholar 

  58. Zhang, Z., Ying, X., Chow, M.-Y.: Decentralizing the economic dispatch problem using a two-level incremental cost consensus algorithm in a smart grid environment. In: 2011 North American Power Symposium, pp. 1–7. IEEE (2011)

    Google Scholar 

  59. Zhang, Z., Chow, M.-Y.: Convergence analysis of the incremental cost consensus algorithm under different communication network topologies in a smart grid. IEEE Trans. Power Syst. 27(4), 1761–1768 (2012)

    Article  Google Scholar 

  60. Kar, S., Hug, G.: Distributed robust economic dispatch in power systems: a consensus + innovations approach. In: Proceedings of IEEE Power Energy Society General Meeting (PESGM), pp. 1–8 (2012)

    Google Scholar 

  61. Hug, G., Kar, S., Wu, C.: Consensus + innovations approach for distributed multiagent coordination in a microgrid. IEEE Trans. Smart Grid 6(4), 1893–1903 (2015)

    Article  Google Scholar 

  62. Loia, V., Vaccaro, A.: Decentralized economic dispatch in smart grids by self-organizing dynamic agents. IEEE Trans. Syst. Man Cybern. Syst. 44(4), 397–408 (2014)

    Google Scholar 

  63. Qin, J., Fu, W., Zheng, W.X., Gao, H.: On the bipartite consensus for generic linear multiagent systems with input saturation. IEEE Trans. Cybern. 47(8), 1948–1958 (2016)

    Article  Google Scholar 

  64. Qin, J., Ma, Q., Gao, H., Shi, Y., Kang, Y.: On group synchronization for interacting clusters of heterogeneous systems. IEEE Trans. Cybern. 47(12), 4122–4133 (2016)

    Article  Google Scholar 

  65. Qin, J., Fu, W., Gao, H., Zheng, W.X.: Distributed k-means algorithm and fuzzy c-means algorithm for sensor networks based on multiagent consensus theory. IEEE Trans. Cybern. 47(3), 772–783 (2016)

    Article  Google Scholar 

  66. Wen, G., Yu, W., Yu, X., Cao, J.: Designing adaptive consensus-based scheme for economic dispatch of smart grid. In: 2016 Eighth International Conference on Advanced Computational Intelligence (ICACI), pp. 236–241. IEEE (2016)

    Google Scholar 

  67. Guo, F., Wen, C., Mao, J., Song, Y.-D.: Distributed economic dispatch for smart grids with random wind power. IEEE Trans. Smart Grid 7(3), 1572–1583 (2015)

    Article  Google Scholar 

  68. Xing, H., Mou, Y., Fu, M., Lin, Z.: Distributed bisection method for economic power dispatch in smart grid. IEEE Trans. Power Syst. 30(6), 3024–3035 (2014)

    Article  Google Scholar 

  69. Yang, T., Wu, D., Sun, Y., Lian, J.: Minimum-time consensus-based approach for power system applications. IEEE Trans. Ind. Electron. 63(2), 1318–1328 (2015)

    Article  Google Scholar 

  70. Wen, G., Yu, X., Liu, Z.-W., Yu, W.: Adaptive consensus-based robust strategy for economic dispatch of smart grids subject to communication uncertainties. IEEE Trans. Ind. Inf. 14(6), 2484–2496 (2017)

    Article  Google Scholar 

  71. Wen, G., Hu, G., Hu, J., Shi, X., Chen, G.: Frequency regulation of source-grid-load systems: a compound control strategy. IEEE Trans. Ind. Inf. 12(1), 69–78 (2015)

    Google Scholar 

  72. Li, C., Yu, X., Huang, T., He, X.: Distributed optimal consensus over resource allocation network and its application to dynamical economic dispatch. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2407–2418 (2017)

    Article  Google Scholar 

  73. Qin, J., Wan, Y., Yu, X., Li, F., Li, C.: Consensus-based distributed coordination between economic dispatch and demand response. IEEE Trans. Smart Grid 10(4), 3709–3719 (2018)

    Article  Google Scholar 

  74. Li, F., Qin, J., Kang, Y.: Multi-agent system based distributed pattern search algorithm for non-convex economic load dispatch in smart grid. IEEE Trans. Power Syst. 34(3), 2093–2102 (2018)

    Article  Google Scholar 

  75. Zhao, C., He, J., Cheng, P., Chen, J.: Analysis of consensus-based distributed economic dispatch under stealthy attacks. IEEE Trans. Ind. Electron. 64(6), 5107–5117 (2016)

    Article  Google Scholar 

  76. Audet, C., Hare, W.: Derivative-Free and Blackbox Optimization, vol. 2. Springer, Berlin (2017)

    Google Scholar 

  77. Durai, S., Subramanian, S., Ganesan, S.: Improved parameters for economic dispatch problems by teaching learning optimization. Int. J. Electr. Power Energy Syst. 67, 11–24 (2015)

    Article  Google Scholar 

  78. Hung, S.-M., Givigi, S.N.: A q-learning approach to flocking with UAVs in a stochastic environment. IEEE Trans. Cybern. 47(1), 186–197 (2016)

    Article  Google Scholar 

  79. Zhang, H., Jiang, H., Luo, Y., Xiao, G.: Data-driven optimal consensus control for discrete-time multi-agent systems with unknown dynamics using reinforcement learning method. IEEE Trans. Ind. Electron. 64(5), 4091–4100 (2016)

    Article  Google Scholar 

  80. Zhang, H., Jiang, H., Luo, C., Xiao, G.: Discrete-time nonzero-sum games for multiplayer using policy-iteration-based adaptive dynamic programming algorithms. IEEE Trans. Cybern. 47(10), 3331–3340 (2016)

    Article  Google Scholar 

  81. Olfati-Saber, R., Murray, R.M.: Consensus problems in networks of agents with switching topology and time-delays. IEEE Trans. Automat. Contr. 49(9), 1520–1533 (2004)

    Article  MATH  Google Scholar 

  82. Saravanan, B., Vasudevan, E., Kothari, D.: Unit commitment problem solution using invasive weed optimization algorithm. Int. J. Electr. Power Energy Syst. 55, 21–28 (2014)

    Article  Google Scholar 

  83. Lujano-Rojas, J., Osório, G., Catalão, J.: New probabilistic method for solving economic dispatch and unit commitment problems incorporating uncertainty due to renewable energy integration. Int. J. Electr. Power Energy Syst. 78, 61–71 (2016)

    Article  Google Scholar 

  84. Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)

    Article  MATH  Google Scholar 

  85. Wang, C., Li, X., Wang, Z., Dong, X., Liang, Z., Liu, X., Liang, J., Han, X.: Day-ahead unit commitment method considering time sequence feature of wind power forecast error. Int. J. Electr. Power Energy Syst. 98, 156–166 (2018)

    Article  Google Scholar 

  86. Carrión, M., Arroyo, J.M.: A computationally efficient mixed-integer linear formulation for the thermal unit commitment problem. IEEE Trans. Power Syst. 21(3), 1371–1378 (2006)

    Article  Google Scholar 

  87. Wang, Y., Zhao, S., Zhou, Z., Botterud, A., Xu, Y., Chen, R.: Risk adjustable day-ahead unit commitment with wind power based on chance constrained goal programming. IEEE Trans. Sustain. Energy 8(2), 530–541 (2016)

    Article  Google Scholar 

  88. Bellifemine, F.L., Caire, G., Greenwood, D.: Developing Multi-agent Systems with JADE. Wiley, New York (2007)

    Google Scholar 

  89. Xu, Y., Zhang, W., Liu, W.: Distributed dynamic programming-based approach for economic dispatch in smart grids. IEEE Trans. Ind. Inf. 11(1), 166–175 (2014)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jiahu Qin .

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Qin, J., Wan, Y., Li, F., Kang, Y., Fu, W. (2023). Model-Free Distributed Optimization. In: Distributed Economic Operation in Smart Grid: Model-Based and Model-Free Perspectives. Studies in Systems, Decision and Control, vol 455. Springer, Singapore. https://doi.org/10.1007/978-981-19-8594-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-19-8594-2_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-19-8593-5

  • Online ISBN: 978-981-19-8594-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics