Abstract
We propose an empirical analysis approach for characterizing tradeoffs between different methods for comparing a set of competing algorithm designs. Our approach can provide insight into performance variation both across candidate algorithms and across instances. It can also identify the best tradeoff between evaluating a larger number of candidate algorithm designs, performing these evaluations on a larger number of problem instances, and allocating more time to each algorithm run. We applied our approach to a study of the rich algorithm design spaces offered by three highly-parameterized, state-of-the-art algorithms for satisfiability and mixed integer programming, considering six different distributions of problem instances. We demonstrate that the resulting algorithm design scenarios differ in many ways, with important consequences for both automatic and manual algorithm design. We expect that both our methods and our findings will lead to tangible improvements in algorithm design methods.
Similar content being viewed by others
References
Adenso-Diaz, B., Laguna, M.: Fine-tuning of algorithms using fractional experimental design and local search. Oper. Res. 54(1), 99–114 (2006)
Ansotegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of solvers. In: Proc. of CP-09, pp. 142–157 (2009)
Audet, C., Orban, D.: Finding optimal algorithmic parameters using the mesh adaptive direct search algorithm. SIAM J. Optim. 17(3), 642–664 (2006)
Babić, D., Hu, A.J.: Structural abstraction of software verification conditions. In: Proc. of CAV-07, pp. 366–378 (2007)
Balaprakash, P., Birattari, M., Stützle, T.: Improvement strategies for the F-Race algorithm: sampling design and iterative refinement. In: Proc. of MH’07, pp. 108–122 (2007)
Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation: the New Experimentalism. Natural Computing Series. Springer Verlag, Berlin (2006)
Beasley, J.: OR-Library: distributing test problems by electronic mail. J. Oper. Res. Soc. 41(11), 1069–1072 (1990)
Birattari, M.: The Problem of Tuning Metaheuristics as Seen from a Machine Learning Perspective. DISKI 292, Infix/Aka, Berlin, Germany (2005)
Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Proc. of GECCO-02, pp. 11–18 (2002)
Etzioni, O., Etzioni, R.: Statistical methods for analyzing speedup learning experiments. J. Mach. Learn. Res. 14(3), 333–347 (1994)
Gent, I.P., Hoos, H.H., Prosser, P., Walsh, T.: Morphing: combining structure and randomness. In: Proc. of AAAI-99, pp. 654–660 (1999)
Gomes, C.P., Selman, B.: Problem structure in the presence of perturbations. In: Proc. of AAAI-97, pp. 221–226 (1997)
Gomes, C.P., Selman, B.: Algorithm portfolios. Artif. Intell. 126(1–2), 43–62 (2001)
Gratch, J., Dejong, G.: Composer: A probabilistic solution to the utility problem in speed-up learning. In: Proc. of AAAI-92, pp. 235–240 (1992)
Hastie, T., Tibshirani, R., Friedman, J.H.: The Elements of Statistical Learning. Springer Series in Statistics. Springer Verlag (2001)
Hoos, H.H.: Computer-aided Design of High-performance Algorithms. Technical Report TR-2008-16, University of British Columbia, Department of Computer Science (2008)
Horvitz, E., Ruan, Y., Gomes, C.P., Kautz, H., Selman, B., Chickering, D.M.: A Bayesian approach to tackling hard computational problems. In: Proc. of UAI-01, pp. 235–244 (2001)
Hutter, F.: Automated configuration of algorithms for solving hard computational problems. Ph.D. thesis, University of British Columbia, Department of Computer Science, Vancouver, Canada (2009)
Hutter, F., Babić, D., Hoos, H.H., Hu, A.J.: Boosting verification by automatic tuning of decision procedures. In: Proc. of FMCAD-07, pp. 27–34 (2007)
Hutter, F., Hamadi, Y., Hoos, H.H., Leyton-Brown, K.: Performance prediction and automated tuning of randomized and parametric algorithms. In: Proc. of CP-06, pp. 213–228 (2006)
Hutter, F., Hoos, H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. JAIR 36, 267–306 (2009)
Hutter, F., Hoos, H.H., Leyton-Brown, K., Murphy, K.P.: An experimental investigation of model-based parameter optimisation: SPO and beyond. In: Proc. of GECCO-09, pp. 271–278 (2009)
Hutter, F., Hoos, H.H., Leyton-Brown, K., Murphy, K.P.: Time-bounded sequential parameter optimization. In: Proc. of LION-10 (2010, to appear)
Hutter, F., Hoos, H.H., Stützle, T.: Automatic algorithm configuration based on local search. In: Proc. of AAAI-07, pp. 1152–1157 (2007)
KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K.: Satenstein: automatically building local search SAT solvers from components. In: Proc. of IJCAI-09, pp. 517–524 (2009)
Leyton-Brown, K., Nudelman, E., Shoham, Y.: Learning the empirical hardness of optimization problems: the case of combinatorial auctions. In: Proc. of CP-02, pp. 556–572 (2002)
Leyton-Brown, K., Nudelman, E., Shoham, Y.: Empirical hardness models: methodology and a case study on combinatorial auctions. J. ACM 56(4), 1–52 (2009)
Leyton-Brown, K., Pearson, M., Shoham, Y.: Towards a universal test suite for combinatorial auction algorithms. In: Proc. of EC-00, pp. 66–76 (2000)
Maron, O., Moore, A.: Hoeffding races: Accelerating model selection search for classification and function approximation. In: Proc. of NIPS-94, pp. 59–66 (1994)
Minton, S.: Automatically configuring constraint satisfaction programs: a case study. Constraints 1(1), 1–40 (1996)
Nudelman, E., Leyton-Brown, K., Hoos, H.H., Devkar, A., Shoham, Y.: Understanding random SAT: beyond the clauses-to-variables ratio. In: Proc. of CP-04 (2004)
Segre, A., Elkan, C., Russell, A.: A critical look at experimental evaluations of EBL. J. Mach. Learn. 6(2), 183–195 (1991)
Simon, L., Chatalic, P.: SATEx: a web-based framework for SAT experimentation. In: Proc. of SAT-01 (2001)
Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: SATzilla: portfolio-based algorithm selection for SAT. J. Artif. Intell. Res. 32, 565–606 (2008)
Zarpas, E.: Benchmarking SAT solvers for bounded model checking. In: Proc. of SAT-05, pp. 340–354 (2005)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hutter, F., Hoos, H.H. & Leyton-Brown, K. Tradeoffs in the empirical evaluation of competing algorithm designs. Ann Math Artif Intell 60, 65–89 (2010). https://doi.org/10.1007/s10472-010-9191-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-010-9191-0