Advertisement

Neural Computing and Applications

, Volume 27, Issue 8, pp 2333–2350 | Cite as

A Q-learning-based swarm optimization algorithm for economic dispatch problem

  • Yi-Zeng Hsieh
  • Mu-Chun SuEmail author
Predictive Analytics Using Machine Learning

Abstract

In this paper, we treat optimization problems as a kind of reinforcement learning problems regarding an optimization procedure for searching an optimal solution as a reinforcement learning procedure for finding the best policy to maximize the expected rewards. This viewpoint motivated us to propose a Q-learning-based swarm optimization (QSO) algorithm. The proposed QSO algorithm is a population-based optimization algorithm which integrates the essential properties of Q-learning and particle swarm optimization. The optimization procedure of the QSO algorithm proceeds as each individual imitates the behavior of the global best one in the swarm. The best individual is chosen based on its accumulated performance instead of its momentary performance at each evaluation. Two data sets including a set of benchmark functions and a real-world problem—the economic dispatch (ED) problem for power systems—were used to test the performance of the proposed QSO algorithm. The simulation results on the benchmark functions show that the proposed QSO algorithm is comparable to or even outperforms several existing optimization algorithms. As for the ED problem, the proposed QSO algorithm has found solutions better than all previously found solutions.

Keywords

Optimization Particle swarm optimization Q-learning Swarm intelligence 

Notes

Acknowledgments

This work was partly supported by Ministry of Science and Technology, Taiwan, under MOST 104-2221-E-008-074-MY2, MOST 103-2911-I-008-001 (support for the Center for Dynamical Biomarkers and Translational Medicine, National Central University, Taiwan).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

References

  1. 1.
    Holland JH (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann ArborGoogle Scholar
  2. 2.
    Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley Publishing Company, ReadingzbMATHGoogle Scholar
  3. 3.
    Fogel LJ (1994) Evolutionary programming in perspective: the top-down view. In: Zurada JM, Marks II RJ, Robinson CJ (eds) Computational intelligence: imitating life. IEEE Press, Piscataway, NJ, pp 135–146Google Scholar
  4. 4.
    Rechenberg I (1965) Cybernetic solution path of an experimental problem, Royal Aircraft Establishment, Library translation, vol 1122. Farnborough, Hants, U.KGoogle Scholar
  5. 5.
    Rechenberg I (1973) Evolutiosstrategie: optimierung technischer system nach prinzipien der biologischen evolution. Frommann-Holzboog Verlag, Stuttgart, GermanyGoogle Scholar
  6. 6.
    Dorigo M, Maniezzo V, Colorni A (1996) Ant system: optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybern B 26(1):29–41CrossRefGoogle Scholar
  7. 7.
    Dorigo M, Stutzle T (2004) Ant colony optimization. MIT, CambridgezbMATHGoogle Scholar
  8. 8.
    Su MC, Zhao YX (2009) A variant of the SOM algorithm and its interpretation in the viewpoint of social influence and learning. Neural Comput Appl 18(8):1043–1055CrossRefGoogle Scholar
  9. 9.
    Chen JH, Su MC, Zhao YX, Hsieh YJ, Chen WH (2008) Application of SOMO based clustering in building renovation. Int J Fuzzy Syst 10(3):195–201Google Scholar
  10. 10.
    Chen JH, Yang LR, Su MC (2009) Comparison of SOM-based optimization and particle swarm optimization for minimizing the construction time of a secant pile wall. Autom Constr 18(6):844–848CrossRefGoogle Scholar
  11. 11.
    Kennedy J, Eberhart R (1995) Particle swarm optimization. In: IEEE international conference on neural networks, vol 4, pp 1942–1948Google Scholar
  12. 12.
    Eberhart R, Kennedy J (1995) A new optimizer using particle swarm theory. In: Proceedings of the sixth international symposium on micro machine and human science, pp 39–43Google Scholar
  13. 13.
    Kennedy J, Eberhart RC, Shi Y (2001) Swarm intelligence. Academic Press, New YorkGoogle Scholar
  14. 14.
    Eberhart RC, Shi Y (1998) Comparison between genetic algorithms and particle swarm optimization. In: Proceedings of 7th international conference of evolutionary programming, pp 611–616Google Scholar
  15. 15.
    Kennedy J (1999) Small worlds and mega-minds: effects of neighborhood topology on particle swarm optimization. In: IEEE international conference on evolutionary computation, pp 1931–1938Google Scholar
  16. 16.
    Clerc M (1999) The swarm and the queen: towards a deterministic and adaptive particle swarm optimization. In: Proceedings of IEEE congress on evolutionary computation, vol 3, pp 1951–1957Google Scholar
  17. 17.
    Løvbjerg M, Rasmussen TK, Krink T (2001) Hybrid particle swarm optimizer with breeding and subpopulations. In: Proceedings of the genetic and evolutionary computation conference, pp 469–476Google Scholar
  18. 18.
    Robinson J, Sinton S, Rahmat-Samii Y (2002) Particle swarm, genetic algorithm, and their hybrids: optimization of a profiled corrugated horn antenna. In: Proceedings of IEEE antennas and propagation society international symposium and URSI National Radio Science Meeting, San Antonio, TX, pp 314–317Google Scholar
  19. 19.
    Kennedy J, Mendes R (2002) Topological structure and particle swarm performance. In: Proceedings of the congress on evolutionary computation, pp 1671–1676Google Scholar
  20. 20.
    Mendes R, Kennedy J, Neves J (2002) Watch thy neighbor or how the swarm can learn from its environment. In: Iberoamerican conference on artificial intelligence, Seville, pp 88–94Google Scholar
  21. 21.
    Balckwell TM, Bentley PJ (2002) Dynamic search with charged swarms. In: Proceedings of the genetic and evolutionary computation conference, pp 19–26Google Scholar
  22. 22.
    Kennedy J, Mendes R (2003) Neighborhood topologies in fully-informed and best-of-neighborhood particle swarms. In: IEEE SMC workshop on soft computing in industrial applications, pp 515–519Google Scholar
  23. 23.
    Esquivel SC, CoelloCoello CA (2003) On the use of particle swarm optimization with multimodal functions. In: Proceedings of IEEE congress on evolutionary computation, vol 2, pp 1130–1136Google Scholar
  24. 24.
    Shi XH, Lu YH, Zhou CG, Lee HP, Lin WZ, Liang YC (2003) Hybrid evolutionary algorithms based on PSO and GA. In: Proceedings of IEEE congress on evolutionary computation, pp 2393–2399Google Scholar
  25. 25.
    Christopher KM, Seppi KD (2004) The Kalman swarm. A new approach to particle motion in swarm optimization. In: Proceedings of GECCO, pp 140–150Google Scholar
  26. 26.
    Devicharan D, Mohan CK (2004) Particle swarm optimization with adaptive linkage learning. In: Proceedings of IEEE congress on evolutionary computation, pp 530–535Google Scholar
  27. 27.
    Settles M, Soule T (2005) Breeding swarms: a GA/PSO hybrid. In: Proceedings of genetic and evolutionary computation conference, pp 161–168Google Scholar
  28. 28.
    Chen X, Li Y (2007) A modified PSO structure resulting in high exploration ability with convergence guaranteed. IEEE Trans Syst Man Cybern B 37(5):1271–1289CrossRefGoogle Scholar
  29. 29.
    Chen YP, Peng WC, Jian MC (2007) Particle swarm optimization with recombination and dynamic linkage discovery. IEEE Trans Syst Man Cybern B 37(6):1460–1470CrossRefGoogle Scholar
  30. 30.
    Kiranyaz S, Ince T, Yildirim A, Gabbouj M (2010) Fractional particle swarm optimization in multi-dimensional search space. IEEE Trans Syst Man Cybern B 40(2):298–319CrossRefGoogle Scholar
  31. 31.
    Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, CambridgeGoogle Scholar
  32. 32.
    Ribeiro C (2002) Reinforcement learning agents. Artif Intell Rev 17:223–250MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4:237–285Google Scholar
  34. 34.
    Barto AG, Sutton RS, Anderson CW (1983) Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans Syst Man Cybern 13(5):834–846CrossRefGoogle Scholar
  35. 35.
    Watkins CJCH, Dayan P (1992) Q-learning machine. Mach Learn 8:279–292zbMATHGoogle Scholar
  36. 36.
    Khajenejad M, Afshinmanesh F, Marandi A, Araabi BN (2006) Intelligent particle swarm optimization using Q-learning. In: Proceedings of IEEE swarm intelligence symposium, pp 7–12Google Scholar
  37. 37.
    Iima H, Kuroe Y (2006) Swarm reinforcement learning algorithm based on exchanging information among agents. Trans Soc Instrum Control Eng 42:1244–1251CrossRefGoogle Scholar
  38. 38.
    De Jong KA (1975) An analysis of the behaviour of a class of genetic adaptive systems. University of Michigan, Ann Arbor (University Microfilms No. 76-9381)Google Scholar
  39. 39.
    Fogel GB, Greenwood GW, Chellapilla K (2000) Evolutionary computation with extinction: experiments and analysis. In: Proceedings of the congress on evolutionary computation, pp 1415–1420Google Scholar
  40. 40.
    Salomon R (1995) Reevaluating genetic algorithm performance under coordinate rotation of benchmark functions. BioSystems 39:263–278CrossRefGoogle Scholar
  41. 41.
    Krink T, Thomsen R (2001) Self-organized criticality and mass extinction in evolutionary algorithms. In: IEEE International conference on evolutionary computation, pp 1155–1161Google Scholar
  42. 42.
    Richards M, Ventura D (2003) Dynamic sociometry in particle swarm optimization. In: International conference on computational intelligence and natural computing, pp 1557–1560Google Scholar
  43. 43.
    Shi Y, Eberhart R (2001) Fuzzy adaptive particle swarm optimization. In: Proceedings of the congress on evolutionary computation, pp 101–106Google Scholar
  44. 44.
    Vesterstrom J, Thomsen R (2004) A comparative study of differential evolution, particle swarm optimization, and evolutionary algorithms on numerical benchmark problems. In: Proceedings of the congress on evolutionary computation, vol 2, pp 1980–1987Google Scholar
  45. 45.
    Brest J, Greiner S, Boskovic B, Mernik M, Zumer V (2006) Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems. In: IEEE transactions on evolutionary computationGoogle Scholar
  46. 46.
    Zhan ZH, Zhan JL, Li Y, Shi YH (2010) Orthogonal learning particle swarm optimization. IEEE Trans Evol Comput 15:1–16CrossRefGoogle Scholar
  47. 47.
    Oca MAM, Stutzle T, Van den Enden K, Dorigo M (2011) Incremental social learning in particle swarms. IEEE Trans Syst Man Cybern B 41(2):368–384CrossRefGoogle Scholar
  48. 48.
    Gao H, Xu W (2011) A new particle swarm algorithm and its globally convergent modifications. IEEE Trans Syst Man Cybern B 41(5):1134–1351MathSciNetGoogle Scholar
  49. 49.
    The software packages for the GAs and the PSO algorithm. www.engr.iupui.edu/~ebergart/web/PSObook.html
  50. 50.
    Tang K, Li X, Suganthan PN, Yang Z, Weise T (2009) Benchmark functions for the CEC’2010 special session and competition on large scale global optimization. Technical Report, Nature Inspired Computation and Applications Laboratory, USTC, China. http://nical.ustc.edu.cn/cec10ss.php
  51. 51.
    Brest J, Zamuda A, Fister I, Maucec MS (2010) Large scale global optimization using self-adaptive differential evolution algorithm evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence, pp 3097–3104, 18–23 July 2010Google Scholar
  52. 52.
    Omidvar MN, Li X, Yao X (2010) Cooperative co-evolution with delta grouping for large scale non-separable function optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence, pp. 1962–1769, 18–23 July 2010Google Scholar
  53. 53.
    Zhao S-Z, Suganthan PN (2010) Dynamic multi-swarm particle swarm optimizer with sub-regional harmony search. In: Swagatam das evolutionary computation, WCCI 2010 IEEE World Congress on Computational Intelligence. pp 1983–1990, 18–23 July 2010Google Scholar
  54. 54.
    Molina D, Lozano M, Herrera F (2010) Memetic algorithm based on local search chains for large scale continuous global optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence. pp 3153–3160, 18–23 July, 2010Google Scholar
  55. 55.
    Korosec P, Tashkova K, Silc J (2010) The differential ant-stigmergy algorithm for large-scale global optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence. pp 4288–4295, 18–23 July 2010Google Scholar
  56. 56.
    Wang H, Wu Z, Rahnamayan S, Jiang D (2010) Sequential DE enhanced by neighborhood search for large scale global optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence, pp 4056–406, 18–23 July 2010Google Scholar
  57. 57.
    Wang Y, Li B (2010) Two-stage based ensemble optimization for large-scale global optimization evolutionary computation. In: WCCI 2010 IEEE world congress on computational intelligence. pp 4488–4495, July 18–23 2010Google Scholar
  58. 58.
    Allen JW, Bruce FW (1984) Power generation, operation, and control. Wiley, New YorkGoogle Scholar
  59. 59.
    Walters DC, Sheble GB (1993) Genetic algorithm solution of economic dispatch with valve point loading. IEEE Trans Power Syst 8(3):1325–1332CrossRefGoogle Scholar
  60. 60.
    Sheble GB, Brittig K (1995) Refined genetic algorithm—economic dispatch example. IEEE Trans Power Syst 10(1):117–124CrossRefGoogle Scholar
  61. 61.
    Chen P-H, Chang H-C (1995) Large-scale economic-dispatch by genetic algorithm. IEEE Trans Power Syst 10(4):1919–1926CrossRefGoogle Scholar
  62. 62.
    Yang H-T, Yang P-C, Huang C-L (1996) Evolutionary programming based economic dispatch for units with non-smooth fuel cost functions. IEEE Trans Power Syst 11(1):112–117CrossRefGoogle Scholar
  63. 63.
    Park YM, Won JR, Park JB (1998) New approach to economic load dispatch based on improved evolutionary programming. Eng Intell Syst Electr Eng Commun 6(2):103–110Google Scholar
  64. 64.
    Wong KP, Yuryevich J (1998) Evolutionary-programming-based algorithm for environmentally-constrained economic dispatch. IEEE Trans Power Syst 13(2):301–306CrossRefGoogle Scholar
  65. 65.
    Yalcinoz T, Altun H, Uzam M (2001) Economic dispatch solution using a genetic algorithm based on arithmetic crossover. In: Proceedings of IEEE power tech conference, vol 2, 10–13 September 2001Google Scholar
  66. 66.
    Lin W-M, Cheng F-S, Tsay M-T (2002) An improved Tabu search for economic dispatch with multiple minima. IEEE Trans Power Syst 17(1):108–112CrossRefGoogle Scholar
  67. 67.
    Sinha N, Chakrabarti R, Chattopadhyay PK (2003) Evolutionary programming technique for economic load dispatch. IEEE Trans Evol Comput 7(1):83–94CrossRefGoogle Scholar
  68. 68.
    Baskar S, Subbaraj P, Rao MVC (2003) Hybrid real coded genetic algorithm solution to economic dispatch problem. Comput Electr Eng 29(3):407–419zbMATHCrossRefGoogle Scholar
  69. 69.
    Gaing ZL (2003) Particle swarm optimization to solving the economic dispatch considering the generator constraints. IEEE Trans Power Syst 18(3):1187–1195CrossRefGoogle Scholar
  70. 70.
    Victoire TAA, Jeyakumar AE (2004) Hybrid PSO-SQP for economic dispatch with valve-point effect. Electr Power Syst Res 71(1):51–59CrossRefGoogle Scholar
  71. 71.
    Victoire TAA, Jeyakumar AE (2004) Particle swarm optimization to solving the economic dispatch considering the generator constraints. IEEE Trans Power Syst 19(4):2121–2122CrossRefGoogle Scholar
  72. 72.
    Park J-B, Lee K-S, Shin J-R, Lee K-Y (2005) A particle swarm optimization for economic dispatch with nonsmooth cost functions. IEEE Trans Power Syst 20(1):34–42CrossRefGoogle Scholar
  73. 73.
    Jayabarathia T, Jayaprakasha K, Jeyakumarb DN, Raghunathan T (2005) Evolutionary programming techniques for different kinds of economic dispatch problems. Electr Power Syst Res 73(2):169–176CrossRefGoogle Scholar

Copyright information

© The Natural Computing Applications Forum 2015

Authors and Affiliations

  1. 1.Department of Technology Application and Human Resource DevelopmentNational Taiwan Normal UniversityTaipeiTaiwan
  2. 2.Department of Computer Science & Information EngineeringNational Central UniversityTaoyuanTaiwan

Personalised recommendations