Benchmarking Evolutionary Algorithms: Towards Exploratory Landscape Analysis

  • Olaf Mersmann
  • Mike Preuss
  • Heike Trautmann
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6238)


We present methods to answer two basic questions that arise when benchmarking optimization algorithms. The first one is: which algorithm is the ‘best’ one? and the second one: which algorithm should I use for my real world problem? Both are connected and neither is easy to answer. We present methods which can be used to analyse the raw data of a benchmark experiment and derive some insight regarding the answers to these questions. We employ the presented methods to analyse the BBOB’09 benchmark results and present some initial findings.


evolutionary optimization benchmarking BBOB test set multidimensional scaling consensus ranking 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    de Borda, J.C.: Mémoire sur les élections au scrutin. Historie de l’Académie Royale des Sciences (1781)Google Scholar
  2. 2.
    Hansen, N., Auger, A., Finck, S., Ros, R.: Real-parameter black-box optimization benchmarking 2009: Experimental setup. Tech. Rep. RR-6828, INRIA (2009),
  3. 3.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, New York (2001)zbMATHGoogle Scholar
  4. 4.
    Hunter, D.J.: Essentials of Discrete Mathematics. Jones and Bartlett Publishers, Boston (2008)Google Scholar
  5. 5.
    Kaufman, L., Rousseeuw, P.: Finding Groups in Data: An Introduction to Cluster Analysis. Wiley Interscience, New York (1990)CrossRefGoogle Scholar
  6. 6.
    Kemeny, J.G., Snell, J.L.: Mathematical Models in the Social Siences. MIT Press, Cambridge (1972)Google Scholar
  7. 7.
    Mersmann, O.: Benchmarking evolutionary multiobjective optimization algorithms using R. Bachelor Thesis, TU Dortmund (2009),
  8. 8.
    Mersmann, O., Trautmann, H., Naujoks, B., Weihs, C.: Benchmarking evolutionary multiobjective optimization algorithms. In: Ishibuchi, H., et al. (eds.) Congress on Evolutionary Computation (CEC). IEEE Press, Piscataway (2010)Google Scholar
  9. 9.
    Saari, D.G.: The optimal ranking method is the Borda Count. Discussion Paper 638, Northwestern University, Center for Mathematical Studies in Economics and Management Science (1985),
  10. 10.
    Saari, D.G., Merlin, V.R.: A geometric examination of Kemeny’s Rule. Social Choice and Welfare 17, 2000 (1997)Google Scholar
  11. 11.
    Suganthan, P.N., Hansen, N., Liang, J.J., Deb, K., Chen, Y.P., Auger, A., Tiwari, S.: Problem definitions and evaluation criteria for the cec 2005 special session on real-parameter optimization. Tech. rep., Nanyang Technological University, Singapore (May 2005),
  12. 12.
    Törn, A., Ali, M., Viitanen, S.: Stochastic Global Optimization: Problem Classes and Solution Techniques. Journal of Global Optimization 14(4), 437–447 (1999)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Olaf Mersmann
    • 1
  • Mike Preuss
    • 1
  • Heike Trautmann
    • 1
  1. 1.TU Dortmund UniversityDortmundGermany

Personalised recommendations