Abstract
It is common to manipulate a large amount of data generated in the benchmarking process when comparing metaheuristics. Performance profiles are analytical tools for the visualization and interpretation of these results. Here we comment on their explanatory power, discuss novel variants, introduce a multicriterion view of the performance comparison, and also define performance profiles for each test-problem in a given benchmark suite. In order to illustrate the potential of performance profiles for both algorithms and test-problems, we apply them to the results of an optimization competition so that new facts are pointed out and conclusions are drawn.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barbosa, H.J.C., Bernardino, H.S., Barreto, A.M.S.: Using performance profiles to analyze the results of the 2006 CEC constrained optimization competition. In: Proceedings of the World Congress on Computational Intelligence – CEC, pp. 1–8. IEEE, Barcelona, Spain (2010)
Barbosa, H.J.C., Bernardino, H.S., Barreto, A.M.S.: Exploring performance proles for analyzing benchmark experiments. In: Proceedings of the Metaheuristics International Conference (MIC), Udine, Italy (2011)
Barr, R., Golden, B., Kelly, J., Resende, M., Stewart, W.: Designing and reporting on computational experiments with heuristic methods. J. Heuristics 1, 9–32 (1995)
Barreto, A.M.S., Bernardino, H.S., Barbosa, H.J.C.: Probabilistic performance profiles for the experimental evaluation of stochastic algorithms. In: Proceedings of the Genetic and Evolutionary Computation Conference – GECCO, pp. 751–758. ACM (2010)
Bartz-Beielstein, T.: Experimental research in evolutionary computation – the new experimentalism. In: Natural Computing Series. Springer, Berlin (2006)
Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.): Experimental Methods for the Analysis of Optimization Algorithms. Springer, Berlin (2010)
Deb, K.: Multi-Objective Optimization Using Evolutionary Algorithms. Wiley-Interscience Series in Systems and Optimization. Wiley, Chichester (2001)
Dolan, E., Moré, J.J.: Benchmarcking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)
Hansen, N., Auger, A., Ros, R., Finck, S., Pošík, P.: Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009. In: Proceedings of the Conference on Genetic and Evolutionary Computation, pp. 1689–1696, New York, NY, USA (2010)
Hooker, J.: Testing heuristics: We have it all wrong. J. Heuristics 1, 33–42 (1995)
Jackson, R.H.F., Boggs, P.T., Nash, S.G., Powell, S.: Guidelines for reporting results of computational experiments. Report of the ad hoc committee. Math. Program. 49, 413–425 (1991)
Moré, J.J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)
Rardin, R.L., Uzsoy, R.: Experimental evaluation of heuristic optimization algorithms: a tutorial. J. Heuristics 7, 261–304 (2001)
Suganthan, P.N.: Special session on constrained real-parameter optimization (2006). http://www3.ntu.edu.sg/home/epnsugan/index_files/CEC-06/CEC06.htm
Whitley, L.D., Mathias, K.E., Rana, S.B., Dzubera, J.: Building better test functions. In: Proceedings of the 6th International Conference on Genetic Algorithms, pp. 239–247. Morgan Kaufmann, San Francisco, CA, USA (1995)
Acknowledgements
The authors thank the reviewers for their comments, and LNCC, CNPq (grant 311651/2006-2), and FAPERJ (grants E-26/ 102.825/2008 and E-26/100.308/2010) for their support.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media New York
About this chapter
Cite this chapter
Barbosa, H.J.C., Bernardino, H.S., Barreto, A.M.S. (2013). Using Performance Profiles for the Analysis and Design of Benchmark Experiments. In: Di Gaspero, L., Schaerf, A., Stützle, T. (eds) Advances in Metaheuristics. Operations Research/Computer Science Interfaces Series, vol 53. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-6322-1_2
Download citation
DOI: https://doi.org/10.1007/978-1-4614-6322-1_2
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-6321-4
Online ISBN: 978-1-4614-6322-1
eBook Packages: Business and EconomicsBusiness and Management (R0)