Optimization Letters

, Volume 1, Issue 3, pp 309–311 | Cite as

How to assess and report the performance of a stochastic algorithm on a benchmark problem: mean or best result on a number of runs?

Short Communication

Abstract

Some authors claim that reporting the best result obtained by a stochastic algorithm in a number of runs is more meaningful than reporting some central statistic. In this short note, we analyze and refute the main argument brought in favor of this statement.

Keywords

Assessment of performance Experimental methodology Stochastic algorithms Metaheuristics 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Eiben, A.E., Jelasity, M.: A critical note on experimental research methodology in EC. In: Proceedings of the 2002 Congress on Evolutionary Computation (CEC’2002), pp. 582–587. IEEE, New York (2002)Google Scholar
  2. 2.
    Eiben A.E., Smith J.E. (2003). Introduction to Evolutionary Computing. Springer, Berlin Heidelberg New YorkMATHGoogle Scholar
  3. 3.
    Fleming P.J., Wallace J.J. (1986). How not to lie with statistics: the correct way to summarize benchmark results. Commun. ACM 29(3):218–221CrossRefGoogle Scholar
  4. 4.
    Barr R.S., Golden B.L., Kelly J.P., Resende M.G.C., Stewart W.R. (1995). Designing and reporting computational experiments with heuristic methods. J. Heuristics 1(1):9–32MATHCrossRefGoogle Scholar
  5. 5.
    Hooker J.N. (1995). Testing heuristics: We have it all wrong. J. Heuristics 1(1):33–42MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Rardin R.R., Uzsoy R. (2001). Experimental evaluation of heuristic optimization algorithms: a tutorial. J. Heuristics 7(2):261–304MATHCrossRefGoogle Scholar
  7. 7.
    Stützle, T.G.: Local Search Algorithms for Combinatorial Problems – Analysis, Algorithms, and New Applications. PhD Thesis, Technische Universität Darmstadt (1999)Google Scholar
  8. 8.
    Glover F., Kochenberger G. (eds) (2002). Handbook of Metaheuristics. Kluwer, NorwellGoogle Scholar
  9. 9.
    Hoos H.H., Stützle T. (2004). Stochastic Local Search. Foundations and Applications. Morgan Kaufmann, San FranciscoGoogle Scholar

Copyright information

© Springer-Verlag 2006

Authors and Affiliations

  1. 1.IRIDIA-CoDEUniversité Libre de BruxellesBrusselsBelgium

Personalised recommendations