Skip to main content
SpringerLink
Log in

Since CEC 2005 competition on real-parameter optimisation: a decade of research, progress and comparative analysis’s weakness

  • Focus
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Real-parameter optimisation is a prolific research line with hundreds of publications per year. There exists an impressive number of alternatives in both algorithm families and enhancements over their respective original proposals. In this work, we analyse if this growth in the number of publications is correlated with a real progress in the field. We have selected five approaches from one of the most significant journals in the field and compared them with the winner of the competition celebrated within the IEEE Congress on Evolutionary Computation 2005. We observe that not only these methods are unable to get the good results of the winner of the competition, published several years before, but that they often avoid this type of comparison. Instead, they usually compare with other approaches from the same family. We conclude that the comparison with the state-of-the-art of the field should be mandatory to promote a real progress and to prevent that the area becomes obfuscated for outsiders.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. http://www.ntu.edu.sg/home/epnsugan/index_files/cec-benchmarking.htm.

  2. http://coco.gforge.inria.fr/.

References

  • Abedinia O, Amjady N, Ghasemi A (2014) A new metaheuristic algorithm based on shark smell optimization. Complexity. doi:10.1002/cplx.21634

    Google Scholar 

  • Addis B, Locatelli M (2007) A new class of test functions for global optimization. J Global Optim 38(3):479–501

    Article  MathSciNet  MATH  Google Scholar 

  • Auger A, Hansen N (2005a) A restart CMA evolution strategy with increasing population size. In: IEEE Congress on Evolutionary Computation (CEC’05), vol 2, pp 1769–1776

  • Auger A, Hansen N (2005b) Performance evaluation of an advanced local search evolutionary algorithm. In: IEEE Congress on Evolutionary Computation (CEC’05), pp 1777–1784

  • Auger A, Hansen N, Schoenauer M (2012) Benchmarking of continuous black box optimization algorithms. Evol Comput 20(4):481–481

    Article  Google Scholar 

  • Awad NH, Ali MZ, Suganthan, PN, Reynolds RG (2016) An ensemble sinusoidal parameter adaptation incorporated with L-SHADE for solving CEC 2014 problems. In: IEEE Congress on Evolutionary Computation (CEC’16), pp 2958–2965

  • Bäck T, Schwefel HP (1993) An overview of evolutionary algorithms for parameter optimization. Evol Comput 1(1):1–23

    Article  Google Scholar 

  • Bersini H, Dorigo M, Langerman S, Seront G, Gambardella L (1996) Results of the first international contest on evolutionary optimisation (1st ICEO), In: IEEE Congress on Evolutionary Computation (CEC’96), pp 611–615

  • Box G (1957) Evolutionary operation: a method for increasing industrial productivity. Appl Stat 6:639–641

    Article  Google Scholar 

  • Bremermann H (1962) Optimization through evolutiona dn recombination. Spartan Books, Washington, pp 93–106

    Google Scholar 

  • Brest J, Greiner S, Boskovic B, Mernik M, Zumer V (2006) Self-adapting control parameters in differential evolution. A comparative study on numerical benchmark problems. IEEE Trans Evol Comput 10(6):646–657

    Article  Google Scholar 

  • Clerc M, Kennedy J (2002) The particle swarm-explosion, stability, and convergence in a multidimensional complex space. IEEE Trans Evol Comput 6(1):58–73

    Article  Google Scholar 

  • Coello CAC (2002) Theoretical and numerical constraint-handling techniques used with evolutionary algorithms: a survey of the state of the art. Comput Methods Appl Mech Eng 191:1245–1287

    Article  MathSciNet  MATH  Google Scholar 

  • Das S, Abraham A, Chakraborty U, Konar A (2009) Differential evolution using a neighbourhood-based mutation operator. IEEE Trans Evol Comput 13(3):526–553

    Article  Google Scholar 

  • de Oca MM, Stützle T, Birattari M, Dorigo M (2009) Frankenstein’s PSO: a composite particle swarm optimization algorithm. IEEE Trans Evol Comput 13(5):1120–1132

    Article  Google Scholar 

  • Deb K, Anand A, Joshi D (2001) A computationally efficient evolutionary algorithm for real-parameter optimization. Evol Comput 9(2):159–195

    Article  Google Scholar 

  • Demsar J (2006) Statistical comparisons of classifers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  • Dorigo M, Stützle T (2004) Ant colony optimization. MIT Press, Cambridge

    MATH  Google Scholar 

  • Dorigo M, Maniezzo V, Colorni A (1996) The ant system: optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybern B Cybern 26(1):29–41

    Article  Google Scholar 

  • Eberhart R, Shi Y (2001) Tracking and optimizing dynamic systems with particle swarms. In: IEEE Congress on Evolutionary Computation (CEC’01), pp 94–100

  • Esbensen H, Mazumder P (1994) SAGA: a unification of the genetic algorithm with simulated annealing and its application to macro-cell placement. In: IEEE Int. Conf. VLSI Des., pp 211–214

  • Eshelman L, Schaffer J (1993) Real-coded genetic algorithms and interval schemata. In: Foundation of Genetic Algorithm-2. Morgan Kaufmann

  • Fogel L (1962) Autonomous automata. Ind Res 4:14–19

    Google Scholar 

  • Fogel DB (2000) Evolutionary computation. Toward a new philosophy of machine intelligence. IEEE Press, Piscataway

    MATH  Google Scholar 

  • Fogel L, Owens A, Walsh M (1966) Artificial intelligence through simulated evolution. Wiley, New York

    MATH  Google Scholar 

  • Friedberg R (1958) A learning machine: Part I. IBM J 2:2–13

    Article  MathSciNet  Google Scholar 

  • Garcia S, Molina D, Lozano M, Herrera F (2009) A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 special session on real parameter optimization. J Heuristics 15(6):617–644

    Article  MATH  Google Scholar 

  • García-Martínez C, Rodriguez FJ, Lozano M (2012) Arbitrary function optimisation with metaheuristics. No free lunch and real-world problems. Soft Comput 16(12):2115–2133

    Article  Google Scholar 

  • Garden RW, Engelbrecht AP (2014) Analysis and classification of optimisation benchmark functions and benchmark suites. In: IEEE Congress on Evolutionary Computation (CEC’2014), pp 1664–1669

  • Geem ZW, Kim JH, Loganathan G (2001) A new heuristic optimization algorithm: harmony search. Simulation 76:60–68

    Article  Google Scholar 

  • Glover F (1977) Heuristics for integer programming using surrogate constraints. Decis Sci 8:156–166

    Article  Google Scholar 

  • Goldberg D (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley, Reading

    MATH  Google Scholar 

  • Guo SM, Yang CC (2015) Enhancing differential evolution utilizing eigenvector-based crossover operator. IEEE Trans Evol Comput 19(1):31–49

    Article  MathSciNet  Google Scholar 

  • Guo SM, Tsai JSH, Yang CC, Hsu PH (2015) A self-optimization approach for L-SHADE incorporated with eigenvector-based crossover and successful-parent-selecting framework on CEC 2015 benchmark set. In: IEEE Congress on Evolutionary Computation (CEC’2015), pp 1003–1010

  • Hansen N (2005) Compilation of results on the CEC benchmark function set. Tech. rep., Institute of Computational Science, ETH Zurich, Switzerland

  • Hansen N (2009) Benchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed. In: Genetic and Evolutionary Computation Conference (GECCO’09), pp 2389–2396

  • Hansen N, Auger A, Mersmann O, Tuv̀ar T, Brockhoff D (2016) COCO: a platform for comparing continuous optimizers in a black-box setting. In: ArXiv e-prints, arXiv:1603.08785

  • Herrera F, Lozano M, Sánchez A (2003) A taxonomy for the crossover operator for real-coded genetic algorithms. An experimental study. Int J Intell Syst 18(3):309–338

    Article  MATH  Google Scholar 

  • Ho SY, Lin HS, Liauh WH, Ho SJ (2008) OPSO: orthogonal particle swarm optimization and its application to task assignment problems. IEEE Trans Syst Man CybernPart A 38(2):288–298

    Google Scholar 

  • Holland J (1962) Outline for a logical theory of adaptive systems. J Assoc Comput Mach 3:297–314

    Article  MATH  Google Scholar 

  • Holland J (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor

    Google Scholar 

  • Jamil M, Yang XS (2013) A literature survey of benchmark functions for global optimisation problems. Int J Math Model Numer Optim 4(2):150–194

    MATH  Google Scholar 

  • Jamil M, Yang X-S, Zepernick H-JD (2013) Test functions for global optimization: a comprehensive survey. In: Swarm Intelligence and Bio-Inspired Computation, pp 193–222

  • Janson S, Middendorf M (2005) A hierarchical particle swarm optimizer and its adaptive variant. IEEE Trans Syst Man Cybern Part B Cybern 35(6):1272–1282

    Article  Google Scholar 

  • Karaboga D, Basturk B (2007) A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm. J Global Optim 39:459–471

    Article  MathSciNet  MATH  Google Scholar 

  • Keenedy J (1999) Small worlds and mega-minds: effects of neighborhood topology on particle swarm performance. In: IEEE Congress on Evolutionary Computation (CEC’99), vol 3, pp 1931–1938

  • Kennedy J, Eberhart R (1995) Particle swarm optimization. IEEE Conf Neural Netw 4:1942–1947

    Google Scholar 

  • Kennedy J, Mendes R (2002) Population structure and particle swarm performance. In: IEEE Congress on Evolutionary Computation (CEC’02), pp 1671–1676

  • KrishnaKumar K, Narayanaswamy S, Garg S (1995) Solving large parameter optimization problems using a genetic algorithm with stochastic coding. In: Genetic Algorithms in Engineering and Computer Science, pp 287–303. Wiley

  • Lee C, Yao X (2004) Evolutionary programming using mutations based on the Lévy probability distribution. IEEE Trans Evol Comput 8(1):1–13

    Article  Google Scholar 

  • Leung YW, Wang Y (2001) An orthogonal genetic algorithm with quantization for global numerical optimization. IEEE Trans Evol Comput 5(1):41–53

    Article  Google Scholar 

  • Li Z (2015) Genetic algorithm that considers scattering for THz quantitative analysis. IEEE Trans Terahertz Sci Technol 5(6):1062–1067

    Article  Google Scholar 

  • Liang J, Suganthan P (2005) Dynamic multi-swarm particle swarm optimizer. In: Swarm Intell. Symposium, pp 124–129

  • Liang JJ, Suganthan PN, Deb K (2005) Novel composition test functions for numerical global optimization. In: IEEE Swarm Intelligence Symposium, pp 68–75

  • Liang J, Qin A, Suganthan P, Baskar S (2006) Comprehensive learning particle swarm optimizer for global optimization of multimodal functions. IEEE Trans Evol Comput 10(3):281–295

    Article  Google Scholar 

  • Liao T, Molina D, de Oca M, Stützle T (2014) A note on bound constraints handling for the IEEE CEC’05 benchmark function suite. Evol Comput 22(2):351–359

    Article  Google Scholar 

  • Liao T, Molina D, Sttzle T (2015) Performance evaluation of automatically tuned continuous optimizers on different benchmark sets. Soft Comput J 27:490–503

    Article  Google Scholar 

  • Liu J, Lampinen (2005) A fuzzy adaptive differential evolution algorithm. Soft Comput 9(6):448–462

    Article  MATH  Google Scholar 

  • Mendes R, Kennedy J, Neves J (2004) The fully informed particle swarm: simpler, maybe better. IEEE Trans Evol Comput 8(3):204–210

    Article  Google Scholar 

  • Omidvar MN, Li X, Tang K (2015) Designing benchmark problems for large-scale continuous optimization. Inf Sci 2015:419–436

    Article  Google Scholar 

  • Omran M, Salman A, Engelbrecht A (2005) Self-adaptive differential evolution. In: Computational Intelligence and Security (LNCS 3801), pp 192–199. Springer

  • Ong YS, Keane A (2004) Meta-lamarckian learning in memetic algorithms. IEEE Trans Evol Comput 8(2):99–110

    Article  Google Scholar 

  • Parsopoulos K, Vrahatis M (2004) UPSO A unified particle swarm optimization scheme. In: Lecture Series on Computational Sciences, pp 868–873

  • Particle Swarm Central (2007) http://www.particleswarm.info/Programs.html#Standard_PSO_2007

  • Passino K (2002) Biomimicry of bacterial foraging for distributed optimization and control. IEEE Control Syst Mag 22:52–67

    Article  Google Scholar 

  • Peram T, Veeramachaneni K, Mohan C (2003) Fitness-distance-ration based particle swarm optimization. In: Swarm Intelligence Symposium, pp 174–181

  • Piotrowski AP (2015) Regarding the rankings of optimization heuristics based on artificially-constructed benchmark functions. Inf Sci 297:191–201

    Article  Google Scholar 

  • Pošic P, Kubalík J (2012) Experimental comparison of six population-based algorithms for continuous black box optimization. Evol Comput 20(4):483–508

    Article  Google Scholar 

  • Pošic P, Huyer W, Pál L (2012) A comparison of global search algorithms for continuous black box optimization. Evol Comput 20(4):509–541

    Article  Google Scholar 

  • Qin A, Huang V, Suganthan P (2009) Differential evolution algorithm with strategy adaptation for global numerical optimization. IEEE Trans Evol Comput 13(2):398–417

    Article  Google Scholar 

  • Qu BY, Liang JJ, Wang ZY, Chen Q, Suganthan PN (2016) Novel benchmark functions for continuous multimodal optimization with comparative results. Swarm Evol Comput 26:23–34

    Article  Google Scholar 

  • Ratnaweera A, Halgamuge S, Watson H (2004) Self-organizing hierarchical particle swarm optimizer with time-varying acceleration coefficients. IEEE Trans Evol Comput 8(3):240–255

    Article  Google Scholar 

  • Rechenberg I (1965) Cybernetic solution path of an experimental problem. Royal Aircraft Establishment Translation, 1122

  • Rönkkönen J, Li X, Kyrki V, Lampinen J (2011) A framework for generating tunable test functions for multimodal optimization. Soft Comput 15(9):1689–1706

    Article  Google Scholar 

  • Schwefel HP (1968) Experimemelle Optimierung einer Zweiphasend. Tech. Rep. 35, Project MHD_Staustrahirohr. 11.034/68

  • Schwefel HP (1975) Evolutionsstrategie und numerische Optimierung. Ph.D. thesis, Technische Universität Berlin

  • Schwefel H-P (1981) Numerical optimization of computer models. Wiley, Chichester

    MATH  Google Scholar 

  • Shi Y, Eberhart R (1998a) A modified particle swarm optimizer. In: IEEE Congress on Evolutionary Computation (CEC’98), pp 69–73

  • Shi Y, Eberhart R (1998b) Parameter selection in particle swarm optimization. In: International Conference on Evolutionary Programming (LNCS 1447), pp 591–600

  • Shi Y, Eberhart R (1999) Empirical study of particle swarm optimization. In: IEEE Congress on Evolutionary Computation (CEC’99), pp 1945–1950

  • Snyman J (1982) A new and dynamic method for unconstrained minimization. Appl Math Model 6:449–462

    Article  MathSciNet  MATH  Google Scholar 

  • Sörensen K (2015) Metaheuristics—the metaphor exposed. Int Trans Oper Res 22:3–18

    Article  MathSciNet  MATH  Google Scholar 

  • Srinivas M, Patnaik L (1994) Adaptive probabilities of crossover and mutation in genetic algorithms. IEEE Trans Syst Man Cybern 24(4):656–667

    Article  Google Scholar 

  • Storn R, Price K (1997) Differential Evolution. A simple and efficient heuristic for global optimization over continuous spaces. J Global Optim 11:341–359

    Article  MathSciNet  MATH  Google Scholar 

  • Suganthan PN, Hansen N, Liang JJ, Deb K, Chen YP, Auger A, Tiwari S (2005) Problem definitions and evaluation criteria for the CEC 2005 special session on real parameter optimization. Tech. report, Nanyang Technological University

  • Tanabe R, Fukunaga A (2013) Success-history based parameter adaptation for differential evolution. In: IEEE Congress on Evolutionary Computation (CEC’13), pp 71–78

  • Tanabe R, Fukunaga A (2014) Improving the search performance of SHADE using linear population size reduction. In: IEEE Congress on Evolutionary Computation (CEC’14), pp 1658–1665

  • Trelea I (2003) The particle swarm optimization algorithm: convergence analysis and parameter selection. Inf Process Lett 85(6):317–325

    Article  MathSciNet  MATH  Google Scholar 

  • van den Bergh F, Engelbrecht A (2004) A cooperative approach to particle swarm optimization. IEEE Trans Evol Comput 8(3):225–239

    Article  Google Scholar 

  • Weyland D (2010) A rigorous analysis of the harmony search algorithm: How the research community can be misled by a novel methodology. Int J Appl Metaheuristic Comput 1(2):50–60

    Article  Google Scholar 

  • Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics 1:80–83

    Article  MathSciNet  Google Scholar 

  • Wolpert D, Macready W (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1(1):67–82

    Article  Google Scholar 

  • Xiong N, Molina D, Ortiz ML, Herrera F (2015) A walk into metaheuristics for engineering optimization: principles, methods and recent trends. Int J Comput Intell Syst 8(4):606–636

    Article  Google Scholar 

  • Yang Z, He J, Yao X (2007) Making a difference to differential evolution. In: Advances Metaheuristics for Hard Optimization, pp 397–414. Springer

  • Yao X, Liu Y, Lin G (1999) Evolutionary programming made faster. IEEE Trans Evol Comput 3(2):82–102

    Article  Google Scholar 

  • Zaharie D (2003) Control of population diversity and adaptation in differential evolution algorithms. In: Mendel 9th Int. Conf. Soft Computing, pp 41–46

  • Zhan ZH, Zhang J, Li Y, Shi YH (2011) Orthogonal learning particle swarm optimization. IEEE Trans Evol Comput 15(6):832–847

    Article  Google Scholar 

  • Zhang J, Sanderson A (2009) JADE: adaptive differential evolution with optional external archive. IEEE Trans Evol Comput 13(5):945–958

    Article  Google Scholar 

  • Zhang Q, Sun J, Tsang E, Ford J (2004) Hybrid estimation of distribution algorithm for global optimization. Eng Comput 21(1):91–107

    Article  MATH  Google Scholar 

  • Zheng YL, Ma LH, Zhang LY, Qian JX (2003a) Empirical study of particle swarm optimizer with an increasing inertia weight. In: IEEE Congress on Evolutionary Computation (CEC’03), pp 221–226

  • Zheng YL, Ma LH, Zhang LY, Qian JX (2003b) On the convergence analysis and parameter selection in particle swarm optimization. In: IEEE International Conference on Machine Learning and Cybernetics, pp 1802–1807

Download references

Acknowledgements

This work was supported by the Research Projects TIN2012-37930-C02-01, TIN2013-47210-P and P12-TIC-2958. P.D. Gutiérrez holds an FPI scholarship from the Spanish Ministry of Economy and Competitiveness (BES-2012-060450).

Author information

Authors and Affiliations

  1. Computing and Numerical Analysis Department, University of Córdoba, 14071, Córdoba, Spain

    Carlos García-Martínez

  2. Department of Computer Science and Artificial Intelligence, University of Granada, 18071, Granada, Spain

    Pablo D. Gutiérrez, Manuel Lozano & Francisco Herrera

  3. Department of Computer Science, University of Cádiz, Cádiz, Spain

    Daniel Molina

Authors
  1. Carlos García-Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pablo D. Gutiérrez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel Molina
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Manuel Lozano
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Francisco Herrera
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlos García-Martínez.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Communicated by C. M. Vide and A. H. Dediu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

García-Martínez, C., Gutiérrez, P.D., Molina, D. et al. Since CEC 2005 competition on real-parameter optimisation: a decade of research, progress and comparative analysis’s weakness. Soft Comput 21, 5573–5583 (2017). https://doi.org/10.1007/s00500-016-2471-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-016-2471-9

Keywords

Search

Navigation