Skip to main content
Log in

Analysing differences between algorithm configurations through ablation

Journal of Heuristics Aims and scope Submit manuscript

Abstract

Developers of high-performance algorithms for hard computational problems increasingly take advantage of automated parameter tuning and algorithm configuration tools, and consequently often create solvers with many parameters and vast configuration spaces. However, there has been very little work to help these algorithm developers answer questions about the high-quality configurations produced by these tools, specifically about which parameter changes contribute most to improved performance. In this work, we present an automated technique for answering such questions by performing ablation analysis between two algorithm configurations. We perform an extensive empirical analysis of our technique on five scenarios from propositional satisfiability, mixed-integer programming and AI planning, and show that in all of these scenarios more than 95 % of the performance gains between default configurations and optimised configurations obtained from automated configuration tools can be explained by modifying the values of a small number of parameters (1–4 in the scenarios we studied). We also investigate the use of our ablation analysis procedure for producing configurations that generalise well to previously-unseen problem domains, as well as for analysing the structure of the algorithm parameter response surface near and between high-performance configurations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Our use of the term ablation follows that of Aghaeepour and Hoos (2013) and loosely echoes its meaning in medicine, where it refers to the surgical removal of organs, organ parts or tissues. We ablate (i.e., remove) changes in the settings of algorithm parameters to better understand the contribution of those changes to observed differences in algorithm performance.

  2. The ablation path illustrated in Fig. 1d resembles the folding pathways observed in biopolymers like RNA that have to overcome a thermodynamic barrier in order to change from one low-energy structure to another (Reidys 2011).

  3. http://www.cs.ubc.ca/labs/beta/Projects/Ablation/.

References

  • Aghaeepour, N., Hoos, H.H.: Ensemble-based prediction of RNA secondary structures. BMC Bioinform. 14, 139 (2013)

    Article  Google Scholar 

  • Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. Proceedings of CP, pp. 142–157. (2009)

  • Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation-The New Experimentalism. Natural Computing Series. Springer, Berlin (2006)

    MATH  Google Scholar 

  • Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Proceedings of GECCO’02, pp. 11–18. (2002)

  • Chiarandini, M., Goegebeur, Y.: Mixed models for the analysis of optimization algorithms. Experimental Methods for the Analysis of Optimization Algorithms, pp. 225–264. (2010)

  • Conover, W.: Practical Nonparametric Statistics, 3rd edn. Wiley, New York (1999)

    Google Scholar 

  • Glover, F.: Genetic algorithms and scatter search: unsuspected potentials. Stat. Comput. 4, 131–140 (1994)

    Article  Google Scholar 

  • Glover, F.: Tabu search and adaptive memory programing—advances, applications and challenges. In: Barr, R.S., Helgason, R.V., Kennington, J.L. (eds.) Interfaces in Computer Science and Operations Research, vol. 7, pp. 1–75. Springer, Nwe York (1997)

    Chapter  Google Scholar 

  • Glover, F., Laguna, M.: Tabu Search. Kluwer Academic Publishers, Boston (1997)

    Book  MATH  Google Scholar 

  • Glover, F., Laguna, M., Martí, R.: Fundamentals of scatter search and path relinking. Control Cybern. 39, 653–684 (2000)

    MathSciNet  MATH  Google Scholar 

  • Glover, F., Laguna, M., Martí, R.: Scatter search and path relinking: advances and applications. In: Glover, F., Kochenberger, G.A. (eds.) Handbook of Metaheuristics, International Series in Operations Research & Management Science, vol. 57, pp. 1–35. Springer, New York (2003)

    Chapter  Google Scholar 

  • Gomes, C.P., van Hoeve, W.J., Sabharwal, A.: Connections in networks: a hybrid approach. Proceedings of CPAIOR , pp. 303–307. (2008)

  • Gunawan, A., Lau, H.C., Lindawati.: Fine-tuning algorithm parameters using the design of experiments approach. In: Proceedings of LION-5, pp. 278–292. (2011)

  • Hooker, G.: Generalized functional ANOVA diagnostics for high dimensional functions of dependent variables. J. Comput. Graph. Stat. 16, 709–732 (2007)

    Article  MathSciNet  Google Scholar 

  • Hoos, H.H.: Programming by optimization. Commun. ACM 55(2), 70–80 (2012)

    Article  Google Scholar 

  • Hutter, F., Babić, D., Hoos, H.H., Hu, A.J.: Boosting verification by automatic tuning of decision procedures. In: Formal Methods in Computer-Aided Design, pp. 27–34. (2007a)

  • Hutter, F., Hoos, H.H., Stützle, T.: Automatic algorithm configuration based on local search. In: AAAI 07, pp. 1152–1157. (2007b)

  • Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intel. Res. 36, 267–306 (2009)

    MATH  Google Scholar 

  • Hutter, F., Hoos, H.H., Leyton-Brown, K.: Automated configuration of mixed integer programming solvers. Proceedings of CPAIOR, pp. 186–202. (2010)

  • Hutter, F., Hoos, H.H., Leyton-Brown, K. :Sequential model-based optimization for general algorithm configuration. In: Proceedings of LION-5, pp. 507–523. (2011)

  • Hutter, F., Hoos, H.H., Leyton-Brown, K.: Identifying key algorithm parameters and instance features using forward selection. In: Proceedings of LION-7, pp. 364–381. (2013)

  • Hutter, F., Hoos, H.H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: Proceedings of the 31st International Conference on Machine Learning (ICML 2014), pp. 754–762. (2014)

  • López-Ibáñez, M., Dubois-Lacoste, J., Stützle, T., Birattari, M.: The irace package, iterated race for automatic algorithm configuration. Tech. Rep. TR/IRIDIA/2011-004, IRIDIA, Université Libre de Bruxelles, Belgium (2011)

  • Maron, O., Moore, A.: Hoeffding races: accelerating model selection search for classification and function approximation. Adv. Neural Inf. Process. Syst. 6, 59–66 (1994)

    Google Scholar 

  • Nannen, V., Eiben, A.: Relevance estimation and value calibration of evolutionary algorithm parameters. Proceedings of IJCAI, pp. 975–980. (2007)

  • Nell, C., Fawcett, C., Hoos, H.H., Leyton-Brown, K.: HAL: A framework for the automated analysis and design of high-performance algorithms. In: Proceedings of LION-5, pp. 600–615. (2011)

  • Reidys, C.M.: Combinatorial Computational Biology of RNA: Pseudoknots and Neutral Networks. Springer, New York (2011)

    Book  MATH  Google Scholar 

  • Styles, J., Hoos, H.H.: Ordered racing protocols for automatically configuring algorithms for scaling performance. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) 2013, pp. 551–558. (2013)

  • Vallati, M., Fawcett, C., Gerevini, A.E., Hoos, H.H., Saetti, A.: Automatic generation of efficient domain-optimized planners from generic parametrized planners. Proceedings of IJCAI RCRA Workshop 2011, pp. 111–123. (2011)

  • Vallati, M., Fawcett, C., Gerevini, A.E., Hoos, H.H., Saetti, A.: Automatic generation of efficient domain-specific planners from generic parametrized planners. In: Proceedings of the 6th Annual Symposium on Combinatorial Search (SoCS-13), pp. 184–192. (2013)

Download references

Acknowledgments

The authors would like to thank our anonymous reviewers, the anonymous reviewers of the earlier version of this paper presented at MIC 2013, and members of the UBC BETA lab for the helpful suggestions and fruitful discussion regarding our approach.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Chris Fawcett.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 185 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fawcett, C., Hoos, H.H. Analysing differences between algorithm configurations through ablation. J Heuristics 22, 431–458 (2016). https://doi.org/10.1007/s10732-014-9275-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10732-014-9275-9

Keywords

Navigation