Optimization Letters

, Volume 1, Issue 3, pp 309–311 | Cite as

How to assess and report the performance of a stochastic algorithm on a benchmark problem: mean or best result on a number of runs?

  • Mauro Birattari
  • Marco Dorigo
Short Communication


Some authors claim that reporting the best result obtained by a stochastic algorithm in a number of runs is more meaningful than reporting some central statistic. In this short note, we analyze and refute the main argument brought in favor of this statement.


Assessment of performance Experimental methodology Stochastic algorithms Metaheuristics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Eiben, A.E., Jelasity, M.: A critical note on experimental research methodology in EC. In: Proceedings of the 2002 Congress on Evolutionary Computation (CEC’2002), pp. 582–587. IEEE, New York (2002)Google Scholar
  2. 2.
    Eiben A.E., Smith J.E. (2003). Introduction to Evolutionary Computing. Springer, Berlin Heidelberg New YorkzbMATHGoogle Scholar
  3. 3.
    Fleming P.J., Wallace J.J. (1986). How not to lie with statistics: the correct way to summarize benchmark results. Commun. ACM 29(3):218–221CrossRefGoogle Scholar
  4. 4.
    Barr R.S., Golden B.L., Kelly J.P., Resende M.G.C., Stewart W.R. (1995). Designing and reporting computational experiments with heuristic methods. J. Heuristics 1(1):9–32zbMATHCrossRefGoogle Scholar
  5. 5.
    Hooker J.N. (1995). Testing heuristics: We have it all wrong. J. Heuristics 1(1):33–42zbMATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Rardin R.R., Uzsoy R. (2001). Experimental evaluation of heuristic optimization algorithms: a tutorial. J. Heuristics 7(2):261–304zbMATHCrossRefGoogle Scholar
  7. 7.
    Stützle, T.G.: Local Search Algorithms for Combinatorial Problems – Analysis, Algorithms, and New Applications. PhD Thesis, Technische Universität Darmstadt (1999)Google Scholar
  8. 8.
    Glover F., Kochenberger G. (eds) (2002). Handbook of Metaheuristics. Kluwer, NorwellGoogle Scholar
  9. 9.
    Hoos H.H., Stützle T. (2004). Stochastic Local Search. Foundations and Applications. Morgan Kaufmann, San FranciscoGoogle Scholar

Copyright information

© Springer-Verlag 2006

Authors and Affiliations

  1. 1.IRIDIA-CoDEUniversité Libre de BruxellesBrusselsBelgium

Personalised recommendations