Skip to main content
Log in

Abstract

We propose an empirical analysis approach for characterizing tradeoffs between different methods for comparing a set of competing algorithm designs. Our approach can provide insight into performance variation both across candidate algorithms and across instances. It can also identify the best tradeoff between evaluating a larger number of candidate algorithm designs, performing these evaluations on a larger number of problem instances, and allocating more time to each algorithm run. We applied our approach to a study of the rich algorithm design spaces offered by three highly-parameterized, state-of-the-art algorithms for satisfiability and mixed integer programming, considering six different distributions of problem instances. We demonstrate that the resulting algorithm design scenarios differ in many ways, with important consequences for both automatic and manual algorithm design. We expect that both our methods and our findings will lead to tangible improvements in algorithm design methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Adenso-Diaz, B., Laguna, M.: Fine-tuning of algorithms using fractional experimental design and local search. Oper. Res. 54(1), 99–114 (2006)

    Article  MATH  Google Scholar 

  2. Ansotegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of solvers. In: Proc. of CP-09, pp. 142–157 (2009)

  3. Audet, C., Orban, D.: Finding optimal algorithmic parameters using the mesh adaptive direct search algorithm. SIAM J. Optim. 17(3), 642–664 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  4. Babić, D., Hu, A.J.: Structural abstraction of software verification conditions. In: Proc. of CAV-07, pp. 366–378 (2007)

  5. Balaprakash, P., Birattari, M., Stützle, T.: Improvement strategies for the F-Race algorithm: sampling design and iterative refinement. In: Proc. of MH’07, pp. 108–122 (2007)

  6. Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation: the New Experimentalism. Natural Computing Series. Springer Verlag, Berlin (2006)

    MATH  Google Scholar 

  7. Beasley, J.: OR-Library: distributing test problems by electronic mail. J. Oper. Res. Soc. 41(11), 1069–1072 (1990)

    Google Scholar 

  8. Birattari, M.: The Problem of Tuning Metaheuristics as Seen from a Machine Learning Perspective. DISKI 292, Infix/Aka, Berlin, Germany (2005)

  9. Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Proc. of GECCO-02, pp. 11–18 (2002)

  10. Etzioni, O., Etzioni, R.: Statistical methods for analyzing speedup learning experiments. J. Mach. Learn. Res. 14(3), 333–347 (1994)

    Google Scholar 

  11. Gent, I.P., Hoos, H.H., Prosser, P., Walsh, T.: Morphing: combining structure and randomness. In: Proc. of AAAI-99, pp. 654–660 (1999)

  12. Gomes, C.P., Selman, B.: Problem structure in the presence of perturbations. In: Proc. of AAAI-97, pp. 221–226 (1997)

  13. Gomes, C.P., Selman, B.: Algorithm portfolios. Artif. Intell. 126(1–2), 43–62 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  14. Gratch, J., Dejong, G.: Composer: A probabilistic solution to the utility problem in speed-up learning. In: Proc. of AAAI-92, pp. 235–240 (1992)

  15. Hastie, T., Tibshirani, R., Friedman, J.H.: The Elements of Statistical Learning. Springer Series in Statistics. Springer Verlag (2001)

  16. Hoos, H.H.: Computer-aided Design of High-performance Algorithms. Technical Report TR-2008-16, University of British Columbia, Department of Computer Science (2008)

  17. Horvitz, E., Ruan, Y., Gomes, C.P., Kautz, H., Selman, B., Chickering, D.M.: A Bayesian approach to tackling hard computational problems. In: Proc. of UAI-01, pp. 235–244 (2001)

  18. Hutter, F.: Automated configuration of algorithms for solving hard computational problems. Ph.D. thesis, University of British Columbia, Department of Computer Science, Vancouver, Canada (2009)

  19. Hutter, F., Babić, D., Hoos, H.H., Hu, A.J.: Boosting verification by automatic tuning of decision procedures. In: Proc. of FMCAD-07, pp. 27–34 (2007)

  20. Hutter, F., Hamadi, Y., Hoos, H.H., Leyton-Brown, K.: Performance prediction and automated tuning of randomized and parametric algorithms. In: Proc. of CP-06, pp. 213–228 (2006)

  21. Hutter, F., Hoos, H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. JAIR 36, 267–306 (2009)

    MATH  Google Scholar 

  22. Hutter, F., Hoos, H.H., Leyton-Brown, K., Murphy, K.P.: An experimental investigation of model-based parameter optimisation: SPO and beyond. In: Proc. of GECCO-09, pp. 271–278 (2009)

  23. Hutter, F., Hoos, H.H., Leyton-Brown, K., Murphy, K.P.: Time-bounded sequential parameter optimization. In: Proc. of LION-10 (2010, to appear)

  24. Hutter, F., Hoos, H.H., Stützle, T.: Automatic algorithm configuration based on local search. In: Proc. of AAAI-07, pp. 1152–1157 (2007)

  25. KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K.: Satenstein: automatically building local search SAT solvers from components. In: Proc. of IJCAI-09, pp. 517–524 (2009)

  26. Leyton-Brown, K., Nudelman, E., Shoham, Y.: Learning the empirical hardness of optimization problems: the case of combinatorial auctions. In: Proc. of CP-02, pp. 556–572 (2002)

  27. Leyton-Brown, K., Nudelman, E., Shoham, Y.: Empirical hardness models: methodology and a case study on combinatorial auctions. J. ACM 56(4), 1–52 (2009)

    Article  MathSciNet  Google Scholar 

  28. Leyton-Brown, K., Pearson, M., Shoham, Y.: Towards a universal test suite for combinatorial auction algorithms. In: Proc. of EC-00, pp. 66–76 (2000)

  29. Maron, O., Moore, A.: Hoeffding races: Accelerating model selection search for classification and function approximation. In: Proc. of NIPS-94, pp. 59–66 (1994)

  30. Minton, S.: Automatically configuring constraint satisfaction programs: a case study. Constraints 1(1), 1–40 (1996)

    Article  MathSciNet  Google Scholar 

  31. Nudelman, E., Leyton-Brown, K., Hoos, H.H., Devkar, A., Shoham, Y.: Understanding random SAT: beyond the clauses-to-variables ratio. In: Proc. of CP-04 (2004)

  32. Segre, A., Elkan, C., Russell, A.: A critical look at experimental evaluations of EBL. J. Mach. Learn. 6(2), 183–195 (1991)

    Google Scholar 

  33. Simon, L., Chatalic, P.: SATEx: a web-based framework for SAT experimentation. In: Proc. of SAT-01 (2001)

  34. Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: SATzilla: portfolio-based algorithm selection for SAT. J. Artif. Intell. Res. 32, 565–606 (2008)

    MATH  Google Scholar 

  35. Zarpas, E.: Benchmarking SAT solvers for bounded model checking. In: Proc. of SAT-05, pp. 340–354 (2005)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Frank Hutter.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hutter, F., Hoos, H.H. & Leyton-Brown, K. Tradeoffs in the empirical evaluation of competing algorithm designs. Ann Math Artif Intell 60, 65–89 (2010). https://doi.org/10.1007/s10472-010-9191-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10472-010-9191-0

Keywords

Mathematics Subject Classifications (2010)

Navigation