Using Performance Profiles for the Analysis and Design of Benchmark Experiments

  • Helio J. C. BarbosaEmail author
  • Heder S. BernardinoEmail author
  • André M. S. BarretoEmail author
Part of the Operations Research/Computer Science Interfaces Series book series (ORCS, volume 53)


It is common to manipulate a large amount of data generated in the benchmarking process when comparing metaheuristics. Performance profiles are analytical tools for the visualization and interpretation of these results. Here we comment on their explanatory power, discuss novel variants, introduce a multicriterion view of the performance comparison, and also define performance profiles for each test-problem in a given benchmark suite. In order to illustrate the potential of performance profiles for both algorithms and test-problems, we apply them to the results of an optimization competition so that new facts are pointed out and conclusions are drawn.



The authors thank the reviewers for their comments, and LNCC, CNPq (grant 311651/2006-2), and FAPERJ (grants E-26/ 102.825/2008 and E-26/100.308/2010) for their support.


  1. 1.
    Barbosa, H.J.C., Bernardino, H.S., Barreto, A.M.S.: Using performance profiles to analyze the results of the 2006 CEC constrained optimization competition. In: Proceedings of the World Congress on Computational Intelligence – CEC, pp. 1–8. IEEE, Barcelona, Spain (2010)Google Scholar
  2. 2.
    Barbosa, H.J.C., Bernardino, H.S., Barreto, A.M.S.: Exploring performance proles for analyzing benchmark experiments. In: Proceedings of the Metaheuristics International Conference (MIC), Udine, Italy (2011)Google Scholar
  3. 3.
    Barr, R., Golden, B., Kelly, J., Resende, M., Stewart, W.: Designing and reporting on computational experiments with heuristic methods. J. Heuristics 1, 9–32 (1995)CrossRefGoogle Scholar
  4. 4.
    Barreto, A.M.S., Bernardino, H.S., Barbosa, H.J.C.: Probabilistic performance profiles for the experimental evaluation of stochastic algorithms. In: Proceedings of the Genetic and Evolutionary Computation Conference – GECCO, pp. 751–758. ACM (2010)Google Scholar
  5. 5.
    Bartz-Beielstein, T.: Experimental research in evolutionary computation – the new experimentalism. In: Natural Computing Series. Springer, Berlin (2006)Google Scholar
  6. 6.
    Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.): Experimental Methods for the Analysis of Optimization Algorithms. Springer, Berlin (2010)Google Scholar
  7. 7.
    Deb, K.: Multi-Objective Optimization Using Evolutionary Algorithms. Wiley-Interscience Series in Systems and Optimization. Wiley, Chichester (2001)Google Scholar
  8. 8.
    Dolan, E., Moré, J.J.: Benchmarcking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)CrossRefGoogle Scholar
  9. 9.
    Hansen, N., Auger, A., Ros, R., Finck, S., Pošík, P.: Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009. In: Proceedings of the Conference on Genetic and Evolutionary Computation, pp. 1689–1696, New York, NY, USA (2010)Google Scholar
  10. 10.
    Hooker, J.: Testing heuristics: We have it all wrong. J. Heuristics 1, 33–42 (1995)CrossRefGoogle Scholar
  11. 11.
    Jackson, R.H.F., Boggs, P.T., Nash, S.G., Powell, S.: Guidelines for reporting results of computational experiments. Report of the ad hoc committee. Math. Program. 49, 413–425 (1991)CrossRefGoogle Scholar
  12. 12.
    Moré, J.J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)CrossRefGoogle Scholar
  13. 13.
    Rardin, R.L., Uzsoy, R.: Experimental evaluation of heuristic optimization algorithms: a tutorial. J. Heuristics 7, 261–304 (2001)CrossRefGoogle Scholar
  14. 14.
    Suganthan, P.N.: Special session on constrained real-parameter optimization (2006).
  15. 15.
    Whitley, L.D., Mathias, K.E., Rana, S.B., Dzubera, J.: Building better test functions. In: Proceedings of the 6th International Conference on Genetic Algorithms, pp. 239–247. Morgan Kaufmann, San Francisco, CA, USA (1995)Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Laboratório Nacional de Computação Científica (LNCC)PetrópolisBrazil
  2. 2.Universidade Federal de Juiz de Fora (UFJF)Juiz de ForaBrazil
  3. 3.School of Computer ScienceMcGill UniversityMontrealCanada

Personalised recommendations