Evaluating Component Solver Contributions to Portfolio-Based Algorithm Selectors

  • Lin Xu
  • Frank Hutter
  • Holger Hoos
  • Kevin Leyton-Brown
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7317)


Portfolio-based methods exploit the complementary strengths of a set of algorithms and—as evidenced in recent competitions—represent the state of the art for solving many NP-hard problems, including SAT. In this work, we argue that a state-of-the-art method for constructing portfolio-based algorithm selectors, \(\texttt{SATzilla}\), also gives rise to an automated method for quantifying the importance of each of a set of available solvers. We entered a substantially improved version of \(\texttt{SATzilla}\) to the inaugural “analysis track” of the 2011 SAT competition, and draw two main conclusions from the results that we obtained. First, automatically-constructed portfolios of sequential, non-portfolio competition entries perform substantially better than the winners of all three sequential categories. Second, and more importantly, a detailed analysis of these portfolios yields valuable insights into the nature of successful solver designs in the different categories. For example, we show that the solvers contributing most to \(\texttt{SATzilla}\) were often not the overall best-performing solvers, but instead solvers that exploit novel solution strategies to solve instances that would remain unsolved without them.


Local Search Marginal Contribution Component Solver Decision Forest Algorithm Portfolio 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Biere, A., Cimatti, A., Clarke, E.M., Fujita, M., Zhu, Y.: Symbolic model checking using SAT procedures instead of BDDs. In: Proc. of DAC 1999, pp. 317–320 (1999)Google Scholar
  2. 2.
    Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001)zbMATHCrossRefGoogle Scholar
  3. 3.
    Carchrae, T., Beck, J.C.: Applying machine learning to low-knowledge control of optimization algorithms. Computational Intelligence 21(4), 372–387 (2005)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Crawford, J.M., Baker, A.B.: Experimental results on the application of satisfiability algorithms to scheduling problems. In: Proc. of AAAI 1994, pp. 1092–1097 (1994)Google Scholar
  5. 5.
    Eén, N., Sörensson, N.: An Extensible SAT-solver. In: Giunchiglia, E., Tacchella, A. (eds.) SAT 2003. LNCS, vol. 2919, pp. 502–518. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  6. 6.
    Gagliolo, M., Schmidhuber, J.: Learning dynamic algorithm portfolios. Annals of Mathematics and Artificial Intelligence 47(3-4), 295–328 (2007)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Gebruers, C., Hnich, B., Bridge, D.G., Freuder, E.C.: Using CBR to Select Solution Strategies in Constraint Programming. In: Muñoz-Ávila, H., Ricci, F. (eds.) ICCBR 2005. LNCS (LNAI), vol. 3620, pp. 222–236. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  8. 8.
    Van Gelder, A., Le Berre, D., Biere, A., Kullmann, O., Simon, L.: Purse-based scoring for comparison of exponential-time programs. In: Proc. of SAT 2005 (2005)Google Scholar
  9. 9.
    Gomes, C.P., Selman, B.: Algorithm portfolios. Artificial Intelligence 126(1-2), 43–62 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Guerri, A., Milano, M.: Learning techniques for automatic algorithm portfolio selection. In: Proc. of ECAI 2004, pp. 475–479 (2004)Google Scholar
  11. 11.
    Helmert, M., Róger, G., Karpas, E.: Fast downward stone soup: A baseline for building planner portfolios. In: Proc. of ICAPS-PAL 2011, pp. 28–35 (2011)Google Scholar
  12. 12.
    Horvitz, E., Ruan, Y., Gomes, C.P., Kautz, H., Selman, B., Chickering, D.M.: A Bayesian approach to tackling hard computational problems. In: Proc. of UAI 2001, pp. 235–244 (2001)Google Scholar
  13. 13.
    Huberman, B.A., Lukose, R.M., Hogg, T.: An economics approach to hard computational problems. Science 265, 51–54 (1997)CrossRefGoogle Scholar
  14. 14.
    Kadioglu, S., Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Algorithm Selection and Scheduling. In: Lee, J. (ed.) CP 2011. LNCS, vol. 6876, pp. 454–469. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  15. 15.
    Kautz, H.A., Selman, B.: Unifying SAT-based and graph-based planning. In: Proc. of IJCAI 1999, pp. 318–325 (1999)Google Scholar
  16. 16.
    Lagoudakis, M.G., Littman, M.L.: Learning to select branching rules in the DPLL procedure for satisfiability. Electronic Notes in Discrete Mathematics, pp. 344–359 (2001)Google Scholar
  17. 17.
    Le Berre, D., Roussel, O., Simon, L.: The international SAT Competitions web page (2012), (last visited on January 29, 2012)
  18. 18.
    Leyton-Brown, K., Nudelman, E., Andrew, G., McFadden, J., Shoham, Y.: A portfolio approach to algorithm selection. In: Proc. of IJCAI 2003, pp. 1542–1543 (2003)Google Scholar
  19. 19.
    Nudelman, E., Leyton-Brown, K., Devkar, A., Shoham, Y., Hoos, H.: Satzilla: An algorithm portfolio for SAT. In: Solver Description, SAT Competition 2004 (2004)Google Scholar
  20. 20.
    Petri, M., Zilberstein, S.: Learning parallel portfolios of algorithms. Annals of Mathematics and Artificial Intelligence 48(1-2), 85–106 (2006)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Rice, J.R.: The algorithm selection problem. Advances in Computers 15, 65–118 (1976)CrossRefGoogle Scholar
  22. 22.
    Roussel, O.: Description of ppfolio (2011),, Solver description (last visited on May 1, 2012)
  23. 23.
    Samulowitz, H., Memisevic, R.: Learning to solve QBF. In: Proc. of AAAI 2007, pp. 255–260 (2007)Google Scholar
  24. 24.
    Smith-Miles, K.: Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Computing Surveys 41(1) (2008)Google Scholar
  25. 25.
    Stephan, P., Brayton, R., Sangiovanni-Vencentelli, A.: Combinational test generation using satisfiability. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 15, 1167–1176 (1996)CrossRefGoogle Scholar
  26. 26.
    Stern, D., Herbrich, R., Graepel, T., Samulowitz, H., Pulina, L., Tacchella, A.: Collaborative expert portfolio management. In: Proc. of AAAI 2010, pp. 210–216 (2010)Google Scholar
  27. 27.
    Streeter, M.J., Smith, S.F.: New techniques for algorithm portfolio design. In: Proc. of UAI 2008, pp. 519–527 (2008)Google Scholar
  28. 28.
    Sutcliffe, G., Suttner, C.B.: Evaluating general purpose automated theorem proving systems. Artificial Intelligence Journal 131(1-2), 39–54 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    Ting, K.M.: An instance-weighting method to induce cost-sensitive trees. IEEE Transactions on Knowledge and Data Engineering 14(3), 659–665 (2002)CrossRefGoogle Scholar
  30. 30.
    van Gelder, A.: Another look at graph coloring via propositional satisfiability. In: Proc. of COLOR 2002, pp. 48–54 (2002)Google Scholar
  31. 31.
    Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: SATzilla: portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research 32, 565–606 (2008)zbMATHGoogle Scholar
  32. 32.
    Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming. In: Proc. of IJCAI-RCRA 2011 (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Lin Xu
    • 1
  • Frank Hutter
    • 1
  • Holger Hoos
    • 1
  • Kevin Leyton-Brown
    • 1
  1. 1.Department of Computer ScienceUniversity of British ColumbiaCanada

Personalised recommendations