Advertisement

Journal of Heuristics

, Volume 22, Issue 4, pp 431–458 | Cite as

Analysing differences between algorithm configurations through ablation

  • Chris Fawcett
  • Holger H. Hoos
Article

Abstract

Developers of high-performance algorithms for hard computational problems increasingly take advantage of automated parameter tuning and algorithm configuration tools, and consequently often create solvers with many parameters and vast configuration spaces. However, there has been very little work to help these algorithm developers answer questions about the high-quality configurations produced by these tools, specifically about which parameter changes contribute most to improved performance. In this work, we present an automated technique for answering such questions by performing ablation analysis between two algorithm configurations. We perform an extensive empirical analysis of our technique on five scenarios from propositional satisfiability, mixed-integer programming and AI planning, and show that in all of these scenarios more than 95 % of the performance gains between default configurations and optimised configurations obtained from automated configuration tools can be explained by modifying the values of a small number of parameters (1–4 in the scenarios we studied). We also investigate the use of our ablation analysis procedure for producing configurations that generalise well to previously-unseen problem domains, as well as for analysing the structure of the algorithm parameter response surface near and between high-performance configurations.

Keywords

Ablation analysis Parameter importance Automated algorithm configuration Empirical analysis 

Notes

Acknowledgments

The authors would like to thank our anonymous reviewers, the anonymous reviewers of the earlier version of this paper presented at MIC 2013, and members of the UBC BETA lab for the helpful suggestions and fruitful discussion regarding our approach.

Supplementary material

10732_2014_9275_MOESM1_ESM.pdf (185 kb)
Supplementary material 1 (pdf 185 KB)

References

  1. Aghaeepour, N., Hoos, H.H.: Ensemble-based prediction of RNA secondary structures. BMC Bioinform. 14, 139 (2013)CrossRefGoogle Scholar
  2. Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. Proceedings of CP, pp. 142–157. (2009)Google Scholar
  3. Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation-The New Experimentalism. Natural Computing Series. Springer, Berlin (2006)zbMATHGoogle Scholar
  4. Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Proceedings of GECCO’02, pp. 11–18. (2002)Google Scholar
  5. Chiarandini, M., Goegebeur, Y.: Mixed models for the analysis of optimization algorithms. Experimental Methods for the Analysis of Optimization Algorithms, pp. 225–264. (2010)Google Scholar
  6. Conover, W.: Practical Nonparametric Statistics, 3rd edn. Wiley, New York (1999)Google Scholar
  7. Glover, F.: Genetic algorithms and scatter search: unsuspected potentials. Stat. Comput. 4, 131–140 (1994)CrossRefGoogle Scholar
  8. Glover, F.: Tabu search and adaptive memory programing—advances, applications and challenges. In: Barr, R.S., Helgason, R.V., Kennington, J.L. (eds.) Interfaces in Computer Science and Operations Research, vol. 7, pp. 1–75. Springer, Nwe York (1997)CrossRefGoogle Scholar
  9. Glover, F., Laguna, M.: Tabu Search. Kluwer Academic Publishers, Boston (1997)CrossRefzbMATHGoogle Scholar
  10. Glover, F., Laguna, M., Martí, R.: Fundamentals of scatter search and path relinking. Control Cybern. 39, 653–684 (2000)MathSciNetzbMATHGoogle Scholar
  11. Glover, F., Laguna, M., Martí, R.: Scatter search and path relinking: advances and applications. In: Glover, F., Kochenberger, G.A. (eds.) Handbook of Metaheuristics, International Series in Operations Research & Management Science, vol. 57, pp. 1–35. Springer, New York (2003)CrossRefGoogle Scholar
  12. Gomes, C.P., van Hoeve, W.J., Sabharwal, A.: Connections in networks: a hybrid approach. Proceedings of CPAIOR , pp. 303–307. (2008)Google Scholar
  13. Gunawan, A., Lau, H.C., Lindawati.: Fine-tuning algorithm parameters using the design of experiments approach. In: Proceedings of LION-5, pp. 278–292. (2011)Google Scholar
  14. Hooker, G.: Generalized functional ANOVA diagnostics for high dimensional functions of dependent variables. J. Comput. Graph. Stat. 16, 709–732 (2007)MathSciNetCrossRefGoogle Scholar
  15. Hoos, H.H.: Programming by optimization. Commun. ACM 55(2), 70–80 (2012)CrossRefGoogle Scholar
  16. Hutter, F., Babić, D., Hoos, H.H., Hu, A.J.: Boosting verification by automatic tuning of decision procedures. In: Formal Methods in Computer-Aided Design, pp. 27–34. (2007a)Google Scholar
  17. Hutter, F., Hoos, H.H., Stützle, T.: Automatic algorithm configuration based on local search. In: AAAI 07, pp. 1152–1157. (2007b)Google Scholar
  18. Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intel. Res. 36, 267–306 (2009)zbMATHGoogle Scholar
  19. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Automated configuration of mixed integer programming solvers. Proceedings of CPAIOR, pp. 186–202. (2010)Google Scholar
  20. Hutter, F., Hoos, H.H., Leyton-Brown, K. :Sequential model-based optimization for general algorithm configuration. In: Proceedings of LION-5, pp. 507–523. (2011)Google Scholar
  21. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Identifying key algorithm parameters and instance features using forward selection. In: Proceedings of LION-7, pp. 364–381. (2013)Google Scholar
  22. Hutter, F., Hoos, H.H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: Proceedings of the 31st International Conference on Machine Learning (ICML 2014), pp. 754–762. (2014)Google Scholar
  23. López-Ibáñez, M., Dubois-Lacoste, J., Stützle, T., Birattari, M.: The irace package, iterated race for automatic algorithm configuration. Tech. Rep. TR/IRIDIA/2011-004, IRIDIA, Université Libre de Bruxelles, Belgium (2011)Google Scholar
  24. Maron, O., Moore, A.: Hoeffding races: accelerating model selection search for classification and function approximation. Adv. Neural Inf. Process. Syst. 6, 59–66 (1994)Google Scholar
  25. Nannen, V., Eiben, A.: Relevance estimation and value calibration of evolutionary algorithm parameters. Proceedings of IJCAI, pp. 975–980. (2007)Google Scholar
  26. Nell, C., Fawcett, C., Hoos, H.H., Leyton-Brown, K.: HAL: A framework for the automated analysis and design of high-performance algorithms. In: Proceedings of LION-5, pp. 600–615. (2011)Google Scholar
  27. Reidys, C.M.: Combinatorial Computational Biology of RNA: Pseudoknots and Neutral Networks. Springer, New York (2011)CrossRefzbMATHGoogle Scholar
  28. Styles, J., Hoos, H.H.: Ordered racing protocols for automatically configuring algorithms for scaling performance. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) 2013, pp. 551–558. (2013)Google Scholar
  29. Vallati, M., Fawcett, C., Gerevini, A.E., Hoos, H.H., Saetti, A.: Automatic generation of efficient domain-optimized planners from generic parametrized planners. Proceedings of IJCAI RCRA Workshop 2011, pp. 111–123. (2011)Google Scholar
  30. Vallati, M., Fawcett, C., Gerevini, A.E., Hoos, H.H., Saetti, A.: Automatic generation of efficient domain-specific planners from generic parametrized planners. In: Proceedings of the 6th Annual Symposium on Combinatorial Search (SoCS-13), pp. 184–192. (2013)Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Computer Science DepartmentUniversity of British ColumbiaVancouverCanada

Personalised recommendations