Abstract
Developers of high-performance algorithms for hard computational problems increasingly take advantage of automated parameter tuning and algorithm configuration tools, and consequently often create solvers with many parameters and vast configuration spaces. However, there has been very little work to help these algorithm developers answer questions about the high-quality configurations produced by these tools, specifically about which parameter changes contribute most to improved performance. In this work, we present an automated technique for answering such questions by performing ablation analysis between two algorithm configurations. We perform an extensive empirical analysis of our technique on five scenarios from propositional satisfiability, mixed-integer programming and AI planning, and show that in all of these scenarios more than 95 % of the performance gains between default configurations and optimised configurations obtained from automated configuration tools can be explained by modifying the values of a small number of parameters (1–4 in the scenarios we studied). We also investigate the use of our ablation analysis procedure for producing configurations that generalise well to previously-unseen problem domains, as well as for analysing the structure of the algorithm parameter response surface near and between high-performance configurations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Our use of the term ablation follows that of Aghaeepour and Hoos (2013) and loosely echoes its meaning in medicine, where it refers to the surgical removal of organs, organ parts or tissues. We ablate (i.e., remove) changes in the settings of algorithm parameters to better understand the contribution of those changes to observed differences in algorithm performance.
References
Aghaeepour, N., Hoos, H.H.: Ensemble-based prediction of RNA secondary structures. BMC Bioinform. 14, 139 (2013)
Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. Proceedings of CP, pp. 142–157. (2009)
Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation-The New Experimentalism. Natural Computing Series. Springer, Berlin (2006)
Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Proceedings of GECCO’02, pp. 11–18. (2002)
Chiarandini, M., Goegebeur, Y.: Mixed models for the analysis of optimization algorithms. Experimental Methods for the Analysis of Optimization Algorithms, pp. 225–264. (2010)
Conover, W.: Practical Nonparametric Statistics, 3rd edn. Wiley, New York (1999)
Glover, F.: Genetic algorithms and scatter search: unsuspected potentials. Stat. Comput. 4, 131–140 (1994)
Glover, F.: Tabu search and adaptive memory programing—advances, applications and challenges. In: Barr, R.S., Helgason, R.V., Kennington, J.L. (eds.) Interfaces in Computer Science and Operations Research, vol. 7, pp. 1–75. Springer, Nwe York (1997)
Glover, F., Laguna, M.: Tabu Search. Kluwer Academic Publishers, Boston (1997)
Glover, F., Laguna, M., Martí, R.: Fundamentals of scatter search and path relinking. Control Cybern. 39, 653–684 (2000)
Glover, F., Laguna, M., Martí, R.: Scatter search and path relinking: advances and applications. In: Glover, F., Kochenberger, G.A. (eds.) Handbook of Metaheuristics, International Series in Operations Research & Management Science, vol. 57, pp. 1–35. Springer, New York (2003)
Gomes, C.P., van Hoeve, W.J., Sabharwal, A.: Connections in networks: a hybrid approach. Proceedings of CPAIOR , pp. 303–307. (2008)
Gunawan, A., Lau, H.C., Lindawati.: Fine-tuning algorithm parameters using the design of experiments approach. In: Proceedings of LION-5, pp. 278–292. (2011)
Hooker, G.: Generalized functional ANOVA diagnostics for high dimensional functions of dependent variables. J. Comput. Graph. Stat. 16, 709–732 (2007)
Hoos, H.H.: Programming by optimization. Commun. ACM 55(2), 70–80 (2012)
Hutter, F., Babić, D., Hoos, H.H., Hu, A.J.: Boosting verification by automatic tuning of decision procedures. In: Formal Methods in Computer-Aided Design, pp. 27–34. (2007a)
Hutter, F., Hoos, H.H., Stützle, T.: Automatic algorithm configuration based on local search. In: AAAI 07, pp. 1152–1157. (2007b)
Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intel. Res. 36, 267–306 (2009)
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Automated configuration of mixed integer programming solvers. Proceedings of CPAIOR, pp. 186–202. (2010)
Hutter, F., Hoos, H.H., Leyton-Brown, K. :Sequential model-based optimization for general algorithm configuration. In: Proceedings of LION-5, pp. 507–523. (2011)
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Identifying key algorithm parameters and instance features using forward selection. In: Proceedings of LION-7, pp. 364–381. (2013)
Hutter, F., Hoos, H.H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: Proceedings of the 31st International Conference on Machine Learning (ICML 2014), pp. 754–762. (2014)
López-Ibáñez, M., Dubois-Lacoste, J., Stützle, T., Birattari, M.: The irace package, iterated race for automatic algorithm configuration. Tech. Rep. TR/IRIDIA/2011-004, IRIDIA, Université Libre de Bruxelles, Belgium (2011)
Maron, O., Moore, A.: Hoeffding races: accelerating model selection search for classification and function approximation. Adv. Neural Inf. Process. Syst. 6, 59–66 (1994)
Nannen, V., Eiben, A.: Relevance estimation and value calibration of evolutionary algorithm parameters. Proceedings of IJCAI, pp. 975–980. (2007)
Nell, C., Fawcett, C., Hoos, H.H., Leyton-Brown, K.: HAL: A framework for the automated analysis and design of high-performance algorithms. In: Proceedings of LION-5, pp. 600–615. (2011)
Reidys, C.M.: Combinatorial Computational Biology of RNA: Pseudoknots and Neutral Networks. Springer, New York (2011)
Styles, J., Hoos, H.H.: Ordered racing protocols for automatically configuring algorithms for scaling performance. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) 2013, pp. 551–558. (2013)
Vallati, M., Fawcett, C., Gerevini, A.E., Hoos, H.H., Saetti, A.: Automatic generation of efficient domain-optimized planners from generic parametrized planners. Proceedings of IJCAI RCRA Workshop 2011, pp. 111–123. (2011)
Vallati, M., Fawcett, C., Gerevini, A.E., Hoos, H.H., Saetti, A.: Automatic generation of efficient domain-specific planners from generic parametrized planners. In: Proceedings of the 6th Annual Symposium on Combinatorial Search (SoCS-13), pp. 184–192. (2013)
Acknowledgments
The authors would like to thank our anonymous reviewers, the anonymous reviewers of the earlier version of this paper presented at MIC 2013, and members of the UBC BETA lab for the helpful suggestions and fruitful discussion regarding our approach.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Fawcett, C., Hoos, H.H. Analysing differences between algorithm configurations through ablation. J Heuristics 22, 431–458 (2016). https://doi.org/10.1007/s10732-014-9275-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10732-014-9275-9