AClib: A Benchmark Library for Algorithm Configuration

  • Frank HutterEmail author
  • Manuel López-Ibáñez
  • Chris Fawcett
  • Marius Lindauer
  • Holger H. Hoos
  • Kevin Leyton-Brown
  • Thomas Stützle
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8426)


Modern solvers for hard computational problems often expose parameters that permit customization for high performance on specific instance types. Since it is tedious and time-consuming to manually optimize such highly parameterized algorithms, recent work in the AI literature has developed automated approaches for this algorithm configuration problem [1, 3, 10, 11, 13, 16].


Algorithm Configuration Problem Benchmark Library Specific Instance Type Hard Computational Problems Configuration Procedure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We gratefully acknowledge all authors of algorithms and instance distributions for making their work available (they are cited on the webpage, acknowledged in README files, and will be cited in a future longer version of this paper). We thank Kevin Tierney and Yuri Malitsky for modifying GGA [1] to support AClib’s format; Lin Xu for generating several instance distributions and writing most feature extraction code for SAT and TSP; Adrian Balint and Sam Bayless for contributing SAT benchmark distributions; Mauro Vallati for exposing many new parameters in LPG; the developers of Fast Downward for helping define its configuration space; and Steve Ramage for helping diagnose and fix problems with several wrappers and runsolver. M. Lindauer acknowledges support by DFG project SCHA 550/8-3, and M. López-Ibáñez acknowledges support from a “Crédit Bref Séjour à l’étranger” from the Belgian F.R.S.-FNRS.


  1. 1.
    Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 142–157. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  2. 2.
    Balint, A., Fröhlich, A., Tompkins, D., Hoos, H.: Sparrow 2011. In: Booklet of SAT-2011 Competition (2011)Google Scholar
  3. 3.
    Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Proceedings of GECCO-02, pp. 11–18 (2002)Google Scholar
  4. 4.
    Chiarandini, M., Fawcett, C., Hoos, H.: A modular multiphase heuristic solver for post enrolment course timetabling. In: Proceedings of PATAT-08 (2008)Google Scholar
  5. 5.
    Dubois-Lacoste, J., López-Ibáñez, M., Stützle, T.: A hybrid TP\(+\)PLS algorithm for bi-objective flow-shop scheduling problems. Comput. Oper. Res. 38(8), 1219–1236 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Fawcett, C., Helmert, M., Hoos, H.H., Karpas, E., Röger, G., Seipp, J.: FD-autotune: domain-specific configuration using fast-downward. In: Proceedings of ICAPS-PAL11 (2011)Google Scholar
  7. 7.
    Hutter, F., Babić, D., Hoos, H.H., Hu, A.J.: Boosting verification by automatic tuning of decision procedures. In: Proceedings of FMCAD-07, pp. 27–34 (2007)Google Scholar
  8. 8.
    Hutter, F., Balint, A., Bayless, S., Hoos, H., Leyton-Brown, K.: Configurable SAT solver challenge (CSSC) (2013), riptsize
  9. 9.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Automated configuration of mixed integer programming solvers. In: Lodi, A., Milano, M., Toth, P. (eds.) CPAIOR 2010. LNCS, vol. 6140, pp. 186–202. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  10. 10.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  11. 11.
    Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. JAIR 36, 267–306 (2009)zbMATHGoogle Scholar
  12. 12.
    KhudaBukhsh, A., Xu, L., Hoos, H.H., Leyton-Brown, K.: SATenstein: automatically building local search SAT solvers from components. In: Proceedings of IJCAI-09, pp. 517–524 (2009)Google Scholar
  13. 13.
    López-Ibáñez, M., Dubois-Lacoste, J., Stützle, T., Birattari, M.: The irace package, iterated race for automatic algorithm configuration. Technical report TR/IRIDIA/2011-004, IRIDIA, Université Libre de Bruxelles, Belgium (2011)Google Scholar
  14. 14.
    López-Ibáñez, M., Stützle, T.: The automatic design of multi-objective ant colony optimization algorithms. IEEE Trans. Evol. Comput. 16(6), 861–875 (2012)CrossRefGoogle Scholar
  15. 15.
    López-Ibáñez, M., Stützle, T.: Automatically improving the anytime behaviour of optimisation algorithms. Eur. J. Oper. Res. (2013)Google Scholar
  16. 16.
    Nannen, V., Eiben, A.: Relevance estimation and value calibration of evolutionary algorithm parameters. In: Proc. of IJCAI-07, pp. 975–980 (2007)Google Scholar
  17. 17.
    Roussel, O.: Controlling a solver execution with the runsolver tool. JSAT 7(4), 139–144 (2011)MathSciNetGoogle Scholar
  18. 18.
    Silverthorn, B., Lierler, Y., Schneider, M.: Surviving solver sensitivity: an ASP practitioner’s guide. In: Proceedings of ICLP-LIPICS-12, pp. 164–175 (2012)Google Scholar
  19. 19.
    Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In: Proceedings of KDD-2013, pp. 847–855 (2013)Google Scholar
  20. 20.
    Tompkins, D.A.D., Balint, A., Hoos, H.H.: Captain Jack: new variable selection heuristics in local search for SAT. In: Sakallah, K.A., Simon, L. (eds.) SAT 2011. LNCS, vol. 6695, pp. 302–316. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  21. 21.
    Vallati, M., Fawcett, C., Gerevini, A.E., Hoos, H.H., Saetti, A.: Generating fast domain-optimized planners by automatically configuring a generic parameterised planner. In: Proceedings of ICAPS-PAL11 (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Frank Hutter
    • 1
    Email author
  • Manuel López-Ibáñez
    • 2
  • Chris Fawcett
    • 3
  • Marius Lindauer
    • 4
  • Holger H. Hoos
    • 3
  • Kevin Leyton-Brown
    • 3
  • Thomas Stützle
    • 2
  1. 1.Department of Computer ScienceFreiburg UniversityFreiburgGermany
  2. 2.IRIDIAUniversité Libre de BruxellesBrusselBelgium
  3. 3.Department of Computer ScienceUniversity of British ColumbiaVancouverCanada
  4. 4.Institute of Computer SciencePotsdam UniversityPotsdamGermany

Personalised recommendations