Soft Computing

, Volume 21, Issue 19, pp 5573–5583 | Cite as

Since CEC 2005 competition on real-parameter optimisation: a decade of research, progress and comparative analysis’s weakness

  • Carlos García-Martínez
  • Pablo D. Gutiérrez
  • Daniel Molina
  • Manuel Lozano
  • Francisco Herrera


Real-parameter optimisation is a prolific research line with hundreds of publications per year. There exists an impressive number of alternatives in both algorithm families and enhancements over their respective original proposals. In this work, we analyse if this growth in the number of publications is correlated with a real progress in the field. We have selected five approaches from one of the most significant journals in the field and compared them with the winner of the competition celebrated within the IEEE Congress on Evolutionary Computation 2005. We observe that not only these methods are unable to get the good results of the winner of the competition, published several years before, but that they often avoid this type of comparison. Instead, they usually compare with other approaches from the same family. We conclude that the comparison with the state-of-the-art of the field should be mandatory to promote a real progress and to prevent that the area becomes obfuscated for outsiders.


Real-parameter optimisation Evolutionary algorithms Nature-inspired algorithms IEEE CEC 2005 State-of-the-art Comparison weaknesses 



This work was supported by the Research Projects TIN2012-37930-C02-01, TIN2013-47210-P and P12-TIC-2958. P.D. Gutiérrez holds an FPI scholarship from the Spanish Ministry of Economy and Competitiveness (BES-2012-060450).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Abedinia O, Amjady N, Ghasemi A (2014) A new metaheuristic algorithm based on shark smell optimization. Complexity. doi: 10.1002/cplx.21634 Google Scholar
  2. Addis B, Locatelli M (2007) A new class of test functions for global optimization. J Global Optim 38(3):479–501MathSciNetCrossRefzbMATHGoogle Scholar
  3. Auger A, Hansen N (2005a) A restart CMA evolution strategy with increasing population size. In: IEEE Congress on Evolutionary Computation (CEC’05), vol 2, pp 1769–1776Google Scholar
  4. Auger A, Hansen N (2005b) Performance evaluation of an advanced local search evolutionary algorithm. In: IEEE Congress on Evolutionary Computation (CEC’05), pp 1777–1784Google Scholar
  5. Auger A, Hansen N, Schoenauer M (2012) Benchmarking of continuous black box optimization algorithms. Evol Comput 20(4):481–481CrossRefGoogle Scholar
  6. Awad NH, Ali MZ, Suganthan, PN, Reynolds RG (2016) An ensemble sinusoidal parameter adaptation incorporated with L-SHADE for solving CEC 2014 problems. In: IEEE Congress on Evolutionary Computation (CEC’16), pp 2958–2965Google Scholar
  7. Bäck T, Schwefel HP (1993) An overview of evolutionary algorithms for parameter optimization. Evol Comput 1(1):1–23CrossRefGoogle Scholar
  8. Bersini H, Dorigo M, Langerman S, Seront G, Gambardella L (1996) Results of the first international contest on evolutionary optimisation (1st ICEO), In: IEEE Congress on Evolutionary Computation (CEC’96), pp 611–615Google Scholar
  9. Box G (1957) Evolutionary operation: a method for increasing industrial productivity. Appl Stat 6:639–641CrossRefGoogle Scholar
  10. Bremermann H (1962) Optimization through evolutiona dn recombination. Spartan Books, Washington, pp 93–106Google Scholar
  11. Brest J, Greiner S, Boskovic B, Mernik M, Zumer V (2006) Self-adapting control parameters in differential evolution. A comparative study on numerical benchmark problems. IEEE Trans Evol Comput 10(6):646–657CrossRefGoogle Scholar
  12. Clerc M, Kennedy J (2002) The particle swarm-explosion, stability, and convergence in a multidimensional complex space. IEEE Trans Evol Comput 6(1):58–73CrossRefGoogle Scholar
  13. Coello CAC (2002) Theoretical and numerical constraint-handling techniques used with evolutionary algorithms: a survey of the state of the art. Comput Methods Appl Mech Eng 191:1245–1287MathSciNetCrossRefzbMATHGoogle Scholar
  14. Das S, Abraham A, Chakraborty U, Konar A (2009) Differential evolution using a neighbourhood-based mutation operator. IEEE Trans Evol Comput 13(3):526–553CrossRefGoogle Scholar
  15. de Oca MM, Stützle T, Birattari M, Dorigo M (2009) Frankenstein’s PSO: a composite particle swarm optimization algorithm. IEEE Trans Evol Comput 13(5):1120–1132CrossRefGoogle Scholar
  16. Deb K, Anand A, Joshi D (2001) A computationally efficient evolutionary algorithm for real-parameter optimization. Evol Comput 9(2):159–195CrossRefGoogle Scholar
  17. Demsar J (2006) Statistical comparisons of classifers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetzbMATHGoogle Scholar
  18. Dorigo M, Stützle T (2004) Ant colony optimization. MIT Press, CambridgezbMATHGoogle Scholar
  19. Dorigo M, Maniezzo V, Colorni A (1996) The ant system: optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybern B Cybern 26(1):29–41CrossRefGoogle Scholar
  20. Eberhart R, Shi Y (2001) Tracking and optimizing dynamic systems with particle swarms. In: IEEE Congress on Evolutionary Computation (CEC’01), pp 94–100Google Scholar
  21. Esbensen H, Mazumder P (1994) SAGA: a unification of the genetic algorithm with simulated annealing and its application to macro-cell placement. In: IEEE Int. Conf. VLSI Des., pp 211–214Google Scholar
  22. Eshelman L, Schaffer J (1993) Real-coded genetic algorithms and interval schemata. In: Foundation of Genetic Algorithm-2. Morgan KaufmannGoogle Scholar
  23. Fogel L (1962) Autonomous automata. Ind Res 4:14–19Google Scholar
  24. Fogel DB (2000) Evolutionary computation. Toward a new philosophy of machine intelligence. IEEE Press, PiscatawayzbMATHGoogle Scholar
  25. Fogel L, Owens A, Walsh M (1966) Artificial intelligence through simulated evolution. Wiley, New YorkzbMATHGoogle Scholar
  26. Friedberg R (1958) A learning machine: Part I. IBM J 2:2–13MathSciNetCrossRefGoogle Scholar
  27. Garcia S, Molina D, Lozano M, Herrera F (2009) A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 special session on real parameter optimization. J Heuristics 15(6):617–644CrossRefzbMATHGoogle Scholar
  28. García-Martínez C, Rodriguez FJ, Lozano M (2012) Arbitrary function optimisation with metaheuristics. No free lunch and real-world problems. Soft Comput 16(12):2115–2133CrossRefGoogle Scholar
  29. Garden RW, Engelbrecht AP (2014) Analysis and classification of optimisation benchmark functions and benchmark suites. In: IEEE Congress on Evolutionary Computation (CEC’2014), pp 1664–1669Google Scholar
  30. Geem ZW, Kim JH, Loganathan G (2001) A new heuristic optimization algorithm: harmony search. Simulation 76:60–68CrossRefGoogle Scholar
  31. Glover F (1977) Heuristics for integer programming using surrogate constraints. Decis Sci 8:156–166CrossRefGoogle Scholar
  32. Goldberg D (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley, ReadingzbMATHGoogle Scholar
  33. Guo SM, Yang CC (2015) Enhancing differential evolution utilizing eigenvector-based crossover operator. IEEE Trans Evol Comput 19(1):31–49MathSciNetCrossRefGoogle Scholar
  34. Guo SM, Tsai JSH, Yang CC, Hsu PH (2015) A self-optimization approach for L-SHADE incorporated with eigenvector-based crossover and successful-parent-selecting framework on CEC 2015 benchmark set. In: IEEE Congress on Evolutionary Computation (CEC’2015), pp 1003–1010Google Scholar
  35. Hansen N (2005) Compilation of results on the CEC benchmark function set. Tech. rep., Institute of Computational Science, ETH Zurich, SwitzerlandGoogle Scholar
  36. Hansen N (2009) Benchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed. In: Genetic and Evolutionary Computation Conference (GECCO’09), pp 2389–2396Google Scholar
  37. Hansen N, Auger A, Mersmann O, Tuv̀ar T, Brockhoff D (2016) COCO: a platform for comparing continuous optimizers in a black-box setting. In: ArXiv e-prints, arXiv:1603.08785
  38. Herrera F, Lozano M, Sánchez A (2003) A taxonomy for the crossover operator for real-coded genetic algorithms. An experimental study. Int J Intell Syst 18(3):309–338CrossRefzbMATHGoogle Scholar
  39. Ho SY, Lin HS, Liauh WH, Ho SJ (2008) OPSO: orthogonal particle swarm optimization and its application to task assignment problems. IEEE Trans Syst Man CybernPart A 38(2):288–298Google Scholar
  40. Holland J (1962) Outline for a logical theory of adaptive systems. J Assoc Comput Mach 3:297–314CrossRefzbMATHGoogle Scholar
  41. Holland J (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann ArborGoogle Scholar
  42. Jamil M, Yang XS (2013) A literature survey of benchmark functions for global optimisation problems. Int J Math Model Numer Optim 4(2):150–194zbMATHGoogle Scholar
  43. Jamil M, Yang X-S, Zepernick H-JD (2013) Test functions for global optimization: a comprehensive survey. In: Swarm Intelligence and Bio-Inspired Computation, pp 193–222Google Scholar
  44. Janson S, Middendorf M (2005) A hierarchical particle swarm optimizer and its adaptive variant. IEEE Trans Syst Man Cybern Part B Cybern 35(6):1272–1282CrossRefGoogle Scholar
  45. Karaboga D, Basturk B (2007) A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm. J Global Optim 39:459–471MathSciNetCrossRefzbMATHGoogle Scholar
  46. Keenedy J (1999) Small worlds and mega-minds: effects of neighborhood topology on particle swarm performance. In: IEEE Congress on Evolutionary Computation (CEC’99), vol 3, pp 1931–1938Google Scholar
  47. Kennedy J, Eberhart R (1995) Particle swarm optimization. IEEE Conf Neural Netw 4:1942–1947Google Scholar
  48. Kennedy J, Mendes R (2002) Population structure and particle swarm performance. In: IEEE Congress on Evolutionary Computation (CEC’02), pp 1671–1676Google Scholar
  49. KrishnaKumar K, Narayanaswamy S, Garg S (1995) Solving large parameter optimization problems using a genetic algorithm with stochastic coding. In: Genetic Algorithms in Engineering and Computer Science, pp 287–303. WileyGoogle Scholar
  50. Lee C, Yao X (2004) Evolutionary programming using mutations based on the Lévy probability distribution. IEEE Trans Evol Comput 8(1):1–13CrossRefGoogle Scholar
  51. Leung YW, Wang Y (2001) An orthogonal genetic algorithm with quantization for global numerical optimization. IEEE Trans Evol Comput 5(1):41–53CrossRefGoogle Scholar
  52. Li Z (2015) Genetic algorithm that considers scattering for THz quantitative analysis. IEEE Trans Terahertz Sci Technol 5(6):1062–1067CrossRefGoogle Scholar
  53. Liang J, Suganthan P (2005) Dynamic multi-swarm particle swarm optimizer. In: Swarm Intell. Symposium, pp 124–129Google Scholar
  54. Liang JJ, Suganthan PN, Deb K (2005) Novel composition test functions for numerical global optimization. In: IEEE Swarm Intelligence Symposium, pp 68–75Google Scholar
  55. Liang J, Qin A, Suganthan P, Baskar S (2006) Comprehensive learning particle swarm optimizer for global optimization of multimodal functions. IEEE Trans Evol Comput 10(3):281–295CrossRefGoogle Scholar
  56. Liao T, Molina D, de Oca M, Stützle T (2014) A note on bound constraints handling for the IEEE CEC’05 benchmark function suite. Evol Comput 22(2):351–359CrossRefGoogle Scholar
  57. Liao T, Molina D, Sttzle T (2015) Performance evaluation of automatically tuned continuous optimizers on different benchmark sets. Soft Comput J 27:490–503CrossRefGoogle Scholar
  58. Liu J, Lampinen (2005) A fuzzy adaptive differential evolution algorithm. Soft Comput 9(6):448–462CrossRefzbMATHGoogle Scholar
  59. Mendes R, Kennedy J, Neves J (2004) The fully informed particle swarm: simpler, maybe better. IEEE Trans Evol Comput 8(3):204–210CrossRefGoogle Scholar
  60. Omidvar MN, Li X, Tang K (2015) Designing benchmark problems for large-scale continuous optimization. Inf Sci 2015:419–436CrossRefGoogle Scholar
  61. Omran M, Salman A, Engelbrecht A (2005) Self-adaptive differential evolution. In: Computational Intelligence and Security (LNCS 3801), pp 192–199. SpringerGoogle Scholar
  62. Ong YS, Keane A (2004) Meta-lamarckian learning in memetic algorithms. IEEE Trans Evol Comput 8(2):99–110CrossRefGoogle Scholar
  63. Parsopoulos K, Vrahatis M (2004) UPSO A unified particle swarm optimization scheme. In: Lecture Series on Computational Sciences, pp 868–873Google Scholar
  64. Passino K (2002) Biomimicry of bacterial foraging for distributed optimization and control. IEEE Control Syst Mag 22:52–67CrossRefGoogle Scholar
  65. Peram T, Veeramachaneni K, Mohan C (2003) Fitness-distance-ration based particle swarm optimization. In: Swarm Intelligence Symposium, pp 174–181Google Scholar
  66. Piotrowski AP (2015) Regarding the rankings of optimization heuristics based on artificially-constructed benchmark functions. Inf Sci 297:191–201CrossRefGoogle Scholar
  67. Pošic P, Kubalík J (2012) Experimental comparison of six population-based algorithms for continuous black box optimization. Evol Comput 20(4):483–508CrossRefGoogle Scholar
  68. Pošic P, Huyer W, Pál L (2012) A comparison of global search algorithms for continuous black box optimization. Evol Comput 20(4):509–541CrossRefGoogle Scholar
  69. Qin A, Huang V, Suganthan P (2009) Differential evolution algorithm with strategy adaptation for global numerical optimization. IEEE Trans Evol Comput 13(2):398–417CrossRefGoogle Scholar
  70. Qu BY, Liang JJ, Wang ZY, Chen Q, Suganthan PN (2016) Novel benchmark functions for continuous multimodal optimization with comparative results. Swarm Evol Comput 26:23–34CrossRefGoogle Scholar
  71. Ratnaweera A, Halgamuge S, Watson H (2004) Self-organizing hierarchical particle swarm optimizer with time-varying acceleration coefficients. IEEE Trans Evol Comput 8(3):240–255CrossRefGoogle Scholar
  72. Rechenberg I (1965) Cybernetic solution path of an experimental problem. Royal Aircraft Establishment Translation, 1122Google Scholar
  73. Rönkkönen J, Li X, Kyrki V, Lampinen J (2011) A framework for generating tunable test functions for multimodal optimization. Soft Comput 15(9):1689–1706CrossRefGoogle Scholar
  74. Schwefel HP (1968) Experimemelle Optimierung einer Zweiphasend. Tech. Rep. 35, Project MHD_Staustrahirohr. 11.034/68Google Scholar
  75. Schwefel HP (1975) Evolutionsstrategie und numerische Optimierung. Ph.D. thesis, Technische Universität BerlinGoogle Scholar
  76. Schwefel H-P (1981) Numerical optimization of computer models. Wiley, ChichesterzbMATHGoogle Scholar
  77. Shi Y, Eberhart R (1998a) A modified particle swarm optimizer. In: IEEE Congress on Evolutionary Computation (CEC’98), pp 69–73Google Scholar
  78. Shi Y, Eberhart R (1998b) Parameter selection in particle swarm optimization. In: International Conference on Evolutionary Programming (LNCS 1447), pp 591–600Google Scholar
  79. Shi Y, Eberhart R (1999) Empirical study of particle swarm optimization. In: IEEE Congress on Evolutionary Computation (CEC’99), pp 1945–1950Google Scholar
  80. Snyman J (1982) A new and dynamic method for unconstrained minimization. Appl Math Model 6:449–462MathSciNetCrossRefzbMATHGoogle Scholar
  81. Sörensen K (2015) Metaheuristics—the metaphor exposed. Int Trans Oper Res 22:3–18MathSciNetCrossRefzbMATHGoogle Scholar
  82. Srinivas M, Patnaik L (1994) Adaptive probabilities of crossover and mutation in genetic algorithms. IEEE Trans Syst Man Cybern 24(4):656–667CrossRefGoogle Scholar
  83. Storn R, Price K (1997) Differential Evolution. A simple and efficient heuristic for global optimization over continuous spaces. J Global Optim 11:341–359MathSciNetCrossRefzbMATHGoogle Scholar
  84. Suganthan PN, Hansen N, Liang JJ, Deb K, Chen YP, Auger A, Tiwari S (2005) Problem definitions and evaluation criteria for the CEC 2005 special session on real parameter optimization. Tech. report, Nanyang Technological UniversityGoogle Scholar
  85. Tanabe R, Fukunaga A (2013) Success-history based parameter adaptation for differential evolution. In: IEEE Congress on Evolutionary Computation (CEC’13), pp 71–78Google Scholar
  86. Tanabe R, Fukunaga A (2014) Improving the search performance of SHADE using linear population size reduction. In: IEEE Congress on Evolutionary Computation (CEC’14), pp 1658–1665Google Scholar
  87. Trelea I (2003) The particle swarm optimization algorithm: convergence analysis and parameter selection. Inf Process Lett 85(6):317–325MathSciNetCrossRefzbMATHGoogle Scholar
  88. van den Bergh F, Engelbrecht A (2004) A cooperative approach to particle swarm optimization. IEEE Trans Evol Comput 8(3):225–239CrossRefGoogle Scholar
  89. Weyland D (2010) A rigorous analysis of the harmony search algorithm: How the research community can be misled by a novel methodology. Int J Appl Metaheuristic Comput 1(2):50–60CrossRefGoogle Scholar
  90. Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics 1:80–83MathSciNetCrossRefGoogle Scholar
  91. Wolpert D, Macready W (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1(1):67–82CrossRefGoogle Scholar
  92. Xiong N, Molina D, Ortiz ML, Herrera F (2015) A walk into metaheuristics for engineering optimization: principles, methods and recent trends. Int J Comput Intell Syst 8(4):606–636CrossRefGoogle Scholar
  93. Yang Z, He J, Yao X (2007) Making a difference to differential evolution. In: Advances Metaheuristics for Hard Optimization, pp 397–414. SpringerGoogle Scholar
  94. Yao X, Liu Y, Lin G (1999) Evolutionary programming made faster. IEEE Trans Evol Comput 3(2):82–102CrossRefGoogle Scholar
  95. Zaharie D (2003) Control of population diversity and adaptation in differential evolution algorithms. In: Mendel 9th Int. Conf. Soft Computing, pp 41–46Google Scholar
  96. Zhan ZH, Zhang J, Li Y, Shi YH (2011) Orthogonal learning particle swarm optimization. IEEE Trans Evol Comput 15(6):832–847CrossRefGoogle Scholar
  97. Zhang J, Sanderson A (2009) JADE: adaptive differential evolution with optional external archive. IEEE Trans Evol Comput 13(5):945–958CrossRefGoogle Scholar
  98. Zhang Q, Sun J, Tsang E, Ford J (2004) Hybrid estimation of distribution algorithm for global optimization. Eng Comput 21(1):91–107CrossRefzbMATHGoogle Scholar
  99. Zheng YL, Ma LH, Zhang LY, Qian JX (2003a) Empirical study of particle swarm optimizer with an increasing inertia weight. In: IEEE Congress on Evolutionary Computation (CEC’03), pp 221–226Google Scholar
  100. Zheng YL, Ma LH, Zhang LY, Qian JX (2003b) On the convergence analysis and parameter selection in particle swarm optimization. In: IEEE International Conference on Machine Learning and Cybernetics, pp 1802–1807Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  1. 1.Computing and Numerical Analysis DepartmentUniversity of CórdobaCórdobaSpain
  2. 2.Department of Computer Science and Artificial IntelligenceUniversity of GranadaGranadaSpain
  3. 3.Department of Computer ScienceUniversity of CádizCádizSpain

Personalised recommendations