Analysis of Algorithm Components and Parameters: Some Case Studies

  • Nguyen DangEmail author
  • Patrick De Causmaecker
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11353)


Modern high-performing algorithms are usually highly parameterised, and can be configured either manually or by an automatic algorithm configurator. The algorithm performance dataset obtained after the configuration step can be used to gain insights into how different algorithm parameters influence algorithm performance. This can be done by a number of analysis methods that exploit the idea of learning prediction models from an algorithm performance dataset and then using them for the data analysis on the importance of variables. In this paper, we demonstrate the complementary usage of three methods along this line, namely forward selection, fANOVA and ablation analysis with surrogates on three case studies, each of which represents some special situations that the analyses can fall into. By these examples, we illustrate how to interpret analysis results and discuss the advantage of combining different analysis methods.


Forward selection fANOVA Ablation analysis with surrogates Parameter analysis 



This work is funded by COMEX (Project P7/36), a BELSPO/IAP Programme. The computational resources and services were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government - department EWI. The authors are grateful to Thomas Stützle and the anonymous reviewers for their valuable comments, which help to improve the quality of the paper.


  1. 1.
    Bartz-Beielstein, T., Lasarczyk, C., Preuss, M.: The sequential parameter optimization toolbox. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.) Experimental Methods for the Analysis of Optimization Algorithms, pp. 337–362. Springer, Heidelberg (2010). Scholar
  2. 2.
    Biedenkapp, A., Lindauer, M., Eggensperger, K., Hutter, F., Fawcett, C., Hoos, H.H.: Efficient parameter importance analysis via ablation with surrogates. In: Singh, S.P., Markovitch, A. (eds.) AAAI Conference on Artificial Intelligence. AAAI Press (2017)Google Scholar
  3. 3.
    Chiarandini, M., Goegebeur, Y.: Mixed models for the analysis of optimization algorithms. Exp. Methods Anal. Optim. Algorithms 1, 225 (2010)CrossRefGoogle Scholar
  4. 4.
    Corstjens, J., Caris, A., Depaire, B., Sörensen, K.: A multilevel methodology for analysing metaheuristic algorithms for the VRPTWGoogle Scholar
  5. 5.
    Dang, N., Pérez Cáceres, L., De Causmaecker, P., Stützle, T.: Configuring irace using surrogate configuration benchmarks. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 243–250. ACM (2017)Google Scholar
  6. 6.
    Fawcett, C., Hoos, H.H.: Analysing differences between algorithm configurations through ablation. J. Heuristics 22(4), 431–458 (2016)CrossRefGoogle Scholar
  7. 7.
    Hooker, G.: Generalized functional ANOVA diagnostics for high-dimensional functions of dependent variables. J. Comput. Graph. Stat 16(3), 709–732 (2012)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). Scholar
  9. 9.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Identifying key algorithm parameters and instance features using forward selection. In: Nicosia, G., Pardalos, P. (eds.) LION 2013. LNCS, vol. 7997, pp. 364–381. Springer, Heidelberg (2013). Scholar
  10. 10.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: Proceedings of the 31th International Conference on Machine Learning, vol. 32, pp. 754–762 (2014)Google Scholar
  11. 11.
    Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intell. Res 36, 267–306 (2009). OctCrossRefGoogle Scholar
  12. 12.
    Hutter, F., Xu, L., Hoos, H.H., Leyton-Brown, K.: Algorithm runtime prediction: methods and evaluation. Artif. Intell. 206, 79–111 (2014)MathSciNetCrossRefGoogle Scholar
  13. 13.
  14. 14.
    López-Ibáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Stützle, T., Birattari, M.: The irace package: Iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43–58 (2016)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Stützle, T.: ACOTSP: a software package of various ant colony optimization algorithms applied to the symmetric traveling salesman problem (2002)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceKU Leuven, CODeS & KULAKLeuvenBelgium
  2. 2.School of Computer ScienceUniversity of St AndrewsSt AndrewsUK

Personalised recommendations