Skip to main content
SpringerLink
Log in

A Q-learning-based swarm optimization algorithm for economic dispatch problem

  • Predictive Analytics Using Machine Learning
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In this paper, we treat optimization problems as a kind of reinforcement learning problems regarding an optimization procedure for searching an optimal solution as a reinforcement learning procedure for finding the best policy to maximize the expected rewards. This viewpoint motivated us to propose a Q-learning-based swarm optimization (QSO) algorithm. The proposed QSO algorithm is a population-based optimization algorithm which integrates the essential properties of Q-learning and particle swarm optimization. The optimization procedure of the QSO algorithm proceeds as each individual imitates the behavior of the global best one in the swarm. The best individual is chosen based on its accumulated performance instead of its momentary performance at each evaluation. Two data sets including a set of benchmark functions and a real-world problem—the economic dispatch (ED) problem for power systems—were used to test the performance of the proposed QSO algorithm. The simulation results on the benchmark functions show that the proposed QSO algorithm is comparable to or even outperforms several existing optimization algorithms. As for the ED problem, the proposed QSO algorithm has found solutions better than all previously found solutions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Holland JH (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor

    Google Scholar 

  2. Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley Publishing Company, Reading

    MATH  Google Scholar 

  3. Fogel LJ (1994) Evolutionary programming in perspective: the top-down view. In: Zurada JM, Marks II RJ, Robinson CJ (eds) Computational intelligence: imitating life. IEEE Press, Piscataway, NJ, pp 135–146

  4. Rechenberg I (1965) Cybernetic solution path of an experimental problem, Royal Aircraft Establishment, Library translation, vol 1122. Farnborough, Hants, U.K

  5. Rechenberg I (1973) Evolutiosstrategie: optimierung technischer system nach prinzipien der biologischen evolution. Frommann-Holzboog Verlag, Stuttgart, Germany

    Google Scholar 

  6. Dorigo M, Maniezzo V, Colorni A (1996) Ant system: optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybern B 26(1):29–41

    Article  Google Scholar 

  7. Dorigo M, Stutzle T (2004) Ant colony optimization. MIT, Cambridge

    MATH  Google Scholar 

  8. Su MC, Zhao YX (2009) A variant of the SOM algorithm and its interpretation in the viewpoint of social influence and learning. Neural Comput Appl 18(8):1043–1055

    Article  Google Scholar 

  9. Chen JH, Su MC, Zhao YX, Hsieh YJ, Chen WH (2008) Application of SOMO based clustering in building renovation. Int J Fuzzy Syst 10(3):195–201

    Google Scholar 

  10. Chen JH, Yang LR, Su MC (2009) Comparison of SOM-based optimization and particle swarm optimization for minimizing the construction time of a secant pile wall. Autom Constr 18(6):844–848

    Article  Google Scholar 

  11. Kennedy J, Eberhart R (1995) Particle swarm optimization. In: IEEE international conference on neural networks, vol 4, pp 1942–1948

  12. Eberhart R, Kennedy J (1995) A new optimizer using particle swarm theory. In: Proceedings of the sixth international symposium on micro machine and human science, pp 39–43

  13. Kennedy J, Eberhart RC, Shi Y (2001) Swarm intelligence. Academic Press, New York

    Google Scholar 

  14. Eberhart RC, Shi Y (1998) Comparison between genetic algorithms and particle swarm optimization. In: Proceedings of 7th international conference of evolutionary programming, pp 611–616

  15. Kennedy J (1999) Small worlds and mega-minds: effects of neighborhood topology on particle swarm optimization. In: IEEE international conference on evolutionary computation, pp 1931–1938

  16. Clerc M (1999) The swarm and the queen: towards a deterministic and adaptive particle swarm optimization. In: Proceedings of IEEE congress on evolutionary computation, vol 3, pp 1951–1957

  17. Løvbjerg M, Rasmussen TK, Krink T (2001) Hybrid particle swarm optimizer with breeding and subpopulations. In: Proceedings of the genetic and evolutionary computation conference, pp 469–476

  18. Robinson J, Sinton S, Rahmat-Samii Y (2002) Particle swarm, genetic algorithm, and their hybrids: optimization of a profiled corrugated horn antenna. In: Proceedings of IEEE antennas and propagation society international symposium and URSI National Radio Science Meeting, San Antonio, TX, pp 314–317

  19. Kennedy J, Mendes R (2002) Topological structure and particle swarm performance. In: Proceedings of the congress on evolutionary computation, pp 1671–1676

  20. Mendes R, Kennedy J, Neves J (2002) Watch thy neighbor or how the swarm can learn from its environment. In: Iberoamerican conference on artificial intelligence, Seville, pp 88–94

  21. Balckwell TM, Bentley PJ (2002) Dynamic search with charged swarms. In: Proceedings of the genetic and evolutionary computation conference, pp 19–26

  22. Kennedy J, Mendes R (2003) Neighborhood topologies in fully-informed and best-of-neighborhood particle swarms. In: IEEE SMC workshop on soft computing in industrial applications, pp 515–519

  23. Esquivel SC, CoelloCoello CA (2003) On the use of particle swarm optimization with multimodal functions. In: Proceedings of IEEE congress on evolutionary computation, vol 2, pp 1130–1136

  24. Shi XH, Lu YH, Zhou CG, Lee HP, Lin WZ, Liang YC (2003) Hybrid evolutionary algorithms based on PSO and GA. In: Proceedings of IEEE congress on evolutionary computation, pp 2393–2399

  25. Christopher KM, Seppi KD (2004) The Kalman swarm. A new approach to particle motion in swarm optimization. In: Proceedings of GECCO, pp 140–150

  26. Devicharan D, Mohan CK (2004) Particle swarm optimization with adaptive linkage learning. In: Proceedings of IEEE congress on evolutionary computation, pp 530–535

  27. Settles M, Soule T (2005) Breeding swarms: a GA/PSO hybrid. In: Proceedings of genetic and evolutionary computation conference, pp 161–168

  28. Chen X, Li Y (2007) A modified PSO structure resulting in high exploration ability with convergence guaranteed. IEEE Trans Syst Man Cybern B 37(5):1271–1289

    Article  Google Scholar 

  29. Chen YP, Peng WC, Jian MC (2007) Particle swarm optimization with recombination and dynamic linkage discovery. IEEE Trans Syst Man Cybern B 37(6):1460–1470

    Article  Google Scholar 

  30. Kiranyaz S, Ince T, Yildirim A, Gabbouj M (2010) Fractional particle swarm optimization in multi-dimensional search space. IEEE Trans Syst Man Cybern B 40(2):298–319

    Article  Google Scholar 

  31. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge

    Google Scholar 

  32. Ribeiro C (2002) Reinforcement learning agents. Artif Intell Rev 17:223–250

    Article  MathSciNet  MATH  Google Scholar 

  33. Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4:237–285

    Google Scholar 

  34. Barto AG, Sutton RS, Anderson CW (1983) Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans Syst Man Cybern 13(5):834–846

    Article  Google Scholar 

  35. Watkins CJCH, Dayan P (1992) Q-learning machine. Mach Learn 8:279–292

    MATH  Google Scholar 

  36. Khajenejad M, Afshinmanesh F, Marandi A, Araabi BN (2006) Intelligent particle swarm optimization using Q-learning. In: Proceedings of IEEE swarm intelligence symposium, pp 7–12

  37. Iima H, Kuroe Y (2006) Swarm reinforcement learning algorithm based on exchanging information among agents. Trans Soc Instrum Control Eng 42:1244–1251

    Article  Google Scholar 

  38. De Jong KA (1975) An analysis of the behaviour of a class of genetic adaptive systems. University of Michigan, Ann Arbor (University Microfilms No. 76-9381)

  39. Fogel GB, Greenwood GW, Chellapilla K (2000) Evolutionary computation with extinction: experiments and analysis. In: Proceedings of the congress on evolutionary computation, pp 1415–1420

  40. Salomon R (1995) Reevaluating genetic algorithm performance under coordinate rotation of benchmark functions. BioSystems 39:263–278

    Article  Google Scholar 

  41. Krink T, Thomsen R (2001) Self-organized criticality and mass extinction in evolutionary algorithms. In: IEEE International conference on evolutionary computation, pp 1155–1161

  42. Richards M, Ventura D (2003) Dynamic sociometry in particle swarm optimization. In: International conference on computational intelligence and natural computing, pp 1557–1560

  43. Shi Y, Eberhart R (2001) Fuzzy adaptive particle swarm optimization. In: Proceedings of the congress on evolutionary computation, pp 101–106

  44. Vesterstrom J, Thomsen R (2004) A comparative study of differential evolution, particle swarm optimization, and evolutionary algorithms on numerical benchmark problems. In: Proceedings of the congress on evolutionary computation, vol 2, pp 1980–1987

  45. Brest J, Greiner S, Boskovic B, Mernik M, Zumer V (2006) Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems. In: IEEE transactions on evolutionary computation

  46. Zhan ZH, Zhan JL, Li Y, Shi YH (2010) Orthogonal learning particle swarm optimization. IEEE Trans Evol Comput 15:1–16

    Article  Google Scholar 

  47. Oca MAM, Stutzle T, Van den Enden K, Dorigo M (2011) Incremental social learning in particle swarms. IEEE Trans Syst Man Cybern B 41(2):368–384

    Article  Google Scholar 

  48. Gao H, Xu W (2011) A new particle swarm algorithm and its globally convergent modifications. IEEE Trans Syst Man Cybern B 41(5):1134–1351

    MathSciNet  Google Scholar 

  49. The software packages for the GAs and the PSO algorithm. www.engr.iupui.edu/~ebergart/web/PSObook.html

  50. Tang K, Li X, Suganthan PN, Yang Z, Weise T (2009) Benchmark functions for the CEC’2010 special session and competition on large scale global optimization. Technical Report, Nature Inspired Computation and Applications Laboratory, USTC, China. http://nical.ustc.edu.cn/cec10ss.php

  51. Brest J, Zamuda A, Fister I, Maucec MS (2010) Large scale global optimization using self-adaptive differential evolution algorithm evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence, pp 3097–3104, 18–23 July 2010

  52. Omidvar MN, Li X, Yao X (2010) Cooperative co-evolution with delta grouping for large scale non-separable function optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence, pp. 1962–1769, 18–23 July 2010

  53. Zhao S-Z, Suganthan PN (2010) Dynamic multi-swarm particle swarm optimizer with sub-regional harmony search. In: Swagatam das evolutionary computation, WCCI 2010 IEEE World Congress on Computational Intelligence. pp 1983–1990, 18–23 July 2010

  54. Molina D, Lozano M, Herrera F (2010) Memetic algorithm based on local search chains for large scale continuous global optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence. pp 3153–3160, 18–23 July, 2010

  55. Korosec P, Tashkova K, Silc J (2010) The differential ant-stigmergy algorithm for large-scale global optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence. pp 4288–4295, 18–23 July 2010

  56. Wang H, Wu Z, Rahnamayan S, Jiang D (2010) Sequential DE enhanced by neighborhood search for large scale global optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence, pp 4056–406, 18–23 July 2010

  57. Wang Y, Li B (2010) Two-stage based ensemble optimization for large-scale global optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence. pp 4488–4495, July 18–23 2010

  58. Allen JW, Bruce FW (1984) Power generation, operation, and control. Wiley, New York

    Google Scholar 

  59. Walters DC, Sheble GB (1993) Genetic algorithm solution of economic dispatch with valve point loading. IEEE Trans Power Syst 8(3):1325–1332

    Article  Google Scholar 

  60. Sheble GB, Brittig K (1995) Refined genetic algorithm—economic dispatch example. IEEE Trans Power Syst 10(1):117–124

    Article  Google Scholar 

  61. Chen P-H, Chang H-C (1995) Large-scale economic-dispatch by genetic algorithm. IEEE Trans Power Syst 10(4):1919–1926

    Article  Google Scholar 

  62. Yang H-T, Yang P-C, Huang C-L (1996) Evolutionary programming based economic dispatch for units with non-smooth fuel cost functions. IEEE Trans Power Syst 11(1):112–117

    Article  Google Scholar 

  63. Park YM, Won JR, Park JB (1998) New approach to economic load dispatch based on improved evolutionary programming. Eng Intell Syst Electr Eng Commun 6(2):103–110

    Google Scholar 

  64. Wong KP, Yuryevich J (1998) Evolutionary-programming-based algorithm for environmentally-constrained economic dispatch. IEEE Trans Power Syst 13(2):301–306

    Article  Google Scholar 

  65. Yalcinoz T, Altun H, Uzam M (2001) Economic dispatch solution using a genetic algorithm based on arithmetic crossover. In: Proceedings of IEEE power tech conference, vol 2, 10–13 September 2001

  66. Lin W-M, Cheng F-S, Tsay M-T (2002) An improved Tabu search for economic dispatch with multiple minima. IEEE Trans Power Syst 17(1):108–112

    Article  Google Scholar 

  67. Sinha N, Chakrabarti R, Chattopadhyay PK (2003) Evolutionary programming technique for economic load dispatch. IEEE Trans Evol Comput 7(1):83–94

    Article  Google Scholar 

  68. Baskar S, Subbaraj P, Rao MVC (2003) Hybrid real coded genetic algorithm solution to economic dispatch problem. Comput Electr Eng 29(3):407–419

    Article  MATH  Google Scholar 

  69. Gaing ZL (2003) Particle swarm optimization to solving the economic dispatch considering the generator constraints. IEEE Trans Power Syst 18(3):1187–1195

    Article  Google Scholar 

  70. Victoire TAA, Jeyakumar AE (2004) Hybrid PSO-SQP for economic dispatch with valve-point effect. Electr Power Syst Res 71(1):51–59

    Article  Google Scholar 

  71. Victoire TAA, Jeyakumar AE (2004) Particle swarm optimization to solving the economic dispatch considering the generator constraints. IEEE Trans Power Syst 19(4):2121–2122

    Article  Google Scholar 

  72. Park J-B, Lee K-S, Shin J-R, Lee K-Y (2005) A particle swarm optimization for economic dispatch with nonsmooth cost functions. IEEE Trans Power Syst 20(1):34–42

    Article  Google Scholar 

  73. Jayabarathia T, Jayaprakasha K, Jeyakumarb DN, Raghunathan T (2005) Evolutionary programming techniques for different kinds of economic dispatch problems. Electr Power Syst Res 73(2):169–176

    Article  Google Scholar 

Download references

Acknowledgments

This work was partly supported by Ministry of Science and Technology, Taiwan, under MOST 104-2221-E-008-074-MY2, MOST 103-2911-I-008-001 (support for the Center for Dynamical Biomarkers and Translational Medicine, National Central University, Taiwan).

Author information

Authors and Affiliations

  1. Department of Technology Application and Human Resource Development, National Taiwan Normal University, Taipei, Taiwan

    Yi-Zeng Hsieh

  2. Department of Computer Science & Information Engineering, National Central University, Taoyuan, Taiwan

    Yi-Zeng Hsieh & Mu-Chun Su

Authors
  1. Yi-Zeng Hsieh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mu-Chun Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mu-Chun Su.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hsieh, YZ., Su, MC. A Q-learning-based swarm optimization algorithm for economic dispatch problem. Neural Comput & Applic 27, 2333–2350 (2016). https://doi.org/10.1007/s00521-015-2070-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-015-2070-1

Keywords

Search

Navigation