Advertisement

Evaluating Instance Generators by Configuration

  • Sam Bayless
  • Dave A. D. Tompkins
  • Holger H. Hoos
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8426)

Abstract

The propositional satisfiability problem (SAT) is one of the most prominent and widely studied NP-hard problems. The development of SAT solvers, whether it is carried out manually or through the use of automated design tools such as algorithm configurators, depends substantially on the sets of benchmark instances used for performance evaluation. Since the supply of instances from real-world applications of SAT is limited, and artificial instance distributions such as Uniform Random \(k\)-SAT are known to have markedly different structure, there has been a long-standing interest in instance generators capable of producing ‘realistic’ SAT instances that could be used during development as proxies for real-world instances. However, it is not obvious how to assess the quality of the instances produced by any such generator. We propose a new approach for evaluating the usefulness of an arbitrary set of instances for use as proxies during solver development, and introduce a new metric, \(Q\)-score, to quantify this. We apply our approach on several artificially generated and real-world benchmark sets and quantitatively compare their usefulness for developing competitive SAT solvers.

Keywords

SAT Benchmark sets Instance generation Automated configuration 

Notes

Acknowledgments

This research has been enabled by the use of computing resources provided by WestGrid and Compute/Calcul Canada, and funding provided by the NSERC Canada Graduate Scholarships and Discovery Grants Programs.

References

  1. 1.
    Achlioptas, D., Gomes, C., Kautz, H., Selman, B.: Generating satisfiable problem instances. In: Proceedings of the National Conference on Artificial Intelligence, pp. 256–261. Menlo Park, CA, Cambridge, MA, London, AAAI Press, MIT Press, 1999 (2000)Google Scholar
  2. 2.
    Ansótegui, C., Bonet, M.L., Levy, J.: Towards industrial-like random SAT instances. In: Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), pp. 387–392 (2009)Google Scholar
  3. 3.
    Ansótegui, C., Bonet, M.L., Levy, J., Manyà, F.: Measuring the hardness of SAT instances. In: Proceedings of the Twenty-Third National Conference on Artificial Intelligence (AAAI-08), pp. 222–229 (2008)Google Scholar
  4. 4.
    Babić, D., Hu, A.J.: Structural abstraction of software verification conditions. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 366–378. Springer, Heidelberg (2007) CrossRefGoogle Scholar
  5. 5.
    Babić, D., Hutter, F.: Spear theorem prover. Solver Description, SAT Race 2008 (2008)Google Scholar
  6. 6.
    Armin, B.: Lingeling, Plingeling PicoSAT and PrecoSAT at SAT race. Technical report 10/1, FMV Report Series, Institute for Formal Models and Verification, Johannes Kepler University (2010)Google Scholar
  7. 7.
    Brummayer, R., Lonsing, F., Biere, A.: Automated testing and debugging of SAT and QBF solvers. In: Strichman, O., Szeider, S. (eds.) SAT 2010. LNCS, vol. 6175, pp. 44–57. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  8. 8.
    Burg, S., Kottler, S., Kaufmann, M.: Creating industrial-like SAT instances by clustering and reconstruction. In: Cimatti, A., Sebastiani, R. (eds.) SAT 2012. LNCS, vol. 7317, pp. 471–472. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  9. 9.
    Chvátal, V., Szemerédi, E.: Many hard examples for resolution. J. ACM 35(4), 759–768 (1988)CrossRefzbMATHGoogle Scholar
  10. 10.
    Dequen, G., Dubois, O.: An efficient approach to solving random \(k\)-SAT problems. J. Autom. Reason. 37(4), 261–276 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  11. 11.
    Gomes, C.P., Selman, B., et al.: Problem structure in the presence of perturbations. In: Proceedings of the National Conference on Artificial Intelligence, pp. 221–226. Wiley (1997)Google Scholar
  12. 12.
    Haanpää, H., Järvisalo, M., Kaski, P., Niemelä, I.: Hard satisfiable clause sets for benchmarking equivalence reasoning techniques. J. Satisf. Boolean Model. Comput. 2(1–4), 27–46 (2006)zbMATHGoogle Scholar
  13. 13.
    Heule, M.J.H., van Maaren, H.: Whose side are you on? finding solutions in a biased search-tree. J. Satisf. Boolean Model. Comput. 4, 117–148 (2008)zbMATHGoogle Scholar
  14. 14.
    Hoos, H.H.: Programming by optimization. Commun. ACM 55, 70–80 (2011)CrossRefGoogle Scholar
  15. 15.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 5. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  16. 16.
    Hutter, F., Babić, D., Hoos, H.H., Hu, A.J.: Boosting verification by automatic tuning of decision procedures. In: Proceedings of the Seventh International Conference on Formal Methods in Computer-Aided Design (FMCAD-07), pp. 27–34 (2007)Google Scholar
  17. 17.
    Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intell. Res. 36, 267–306 (2009)zbMATHGoogle Scholar
  18. 18.
    Hutter, F., Hoos, H.H., Stützle, T.: Automatic algorithm configuration based on local search. In: Proceedings of the Twenty-Second National Conference on Artificial Intelligence (AAAI-07), pp. 1152–1157 (2007)Google Scholar
  19. 19.
    Järvisalo, M., Kaski, P., Koivisto, M., Korhonen, J.H.: Finding efficient circuits for ensemble computation. In: Cimatti, A., Sebastiani, R. (eds.) SAT 2012. LNCS, vol. 7317, pp. 369–382. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  20. 20.
    Kautz, H., Selman, B.: Ten challenges Redux: recent progress in propositional reasoning and search. In: Rossi, F. (ed.) CP 2003. LNCS, vol. 2833, pp. 1–18. Springer, Heidelberg (2003) CrossRefGoogle Scholar
  21. 21.
    KhudaBukhsh, A.R.: SATenstein: automatically building local search SAT solvers from components. Master’s thesis, University of British Columbia (2009)Google Scholar
  22. 22.
    KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K.: SATenstein: automatically building local search SAT solvers from components. In: Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), pp. 517–524 (2009)Google Scholar
  23. 23.
    Li, C.M., Anbulagan, : Look-ahead versus look-back for satisfiability problems. In: Smolka, G. (ed.) CP 1997. LNCS, vol. 1330, pp. 341–355. Springer, Heidelberg (1997) CrossRefGoogle Scholar
  24. 24.
    Rish, I., Dechter, R.: Resolution versus search: two strategies for sat. J. Autom. Reason. 24(1), 225–275 (2000)CrossRefzbMATHMathSciNetGoogle Scholar
  25. 25.
    Selman, B., Kautz, H., McAllester, D.: Ten challenges in propositional reasoning and search. In: Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI-97), pp. 50–54 (1997)Google Scholar
  26. 26.
    Slater, A.: Modelling more realistic SAT problems. In: McKay, B., Slaney, J.K. (eds.) AI 2002. LNCS (LNAI), vol. 2557, pp. 591–602. Springer, Heidelberg (2002) Google Scholar
  27. 27.
    Sörensson, N., Eén, N.: Minisat v1.13 - a SAT solver with conflict-clause minimization. In: Poster, Eighth International Conference on Theory and Applications of Satisfiability Testing (SAT-05) (2005)Google Scholar
  28. 28.
    Tompkins, D.A.D., Balint, A., Hoos, H.H.: Captain Jack: new variable selection heuristics in local search for SAT. In: Sakallah, K.A., Simon, L. (eds.) SAT 2011. LNCS, vol. 6695, pp. 302–316. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  29. 29.
    Tompkins, D.A.D., Hoos, H.H.: Dynamic scoring functions with variable expressions: new SLS methods for solving SAT. In: Strichman, O., Szeider, S. (eds.) SAT 2010. LNCS, vol. 6175, pp. 278–292. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  30. 30.
    Zarpas, E.: Benchmarking SAT solvers for bounded model checking. In: Bacchus, F., Walsh, T. (eds.) SAT 2005. LNCS, vol. 3569, pp. 340–354. Springer, Heidelberg (2005) CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Sam Bayless
    • 1
  • Dave A. D. Tompkins
    • 2
  • Holger H. Hoos
    • 1
  1. 1.Department of Computer ScienceUniversity of British ColumbiaVancouverCanada
  2. 2.David R. Cheriton School of Computer ScienceUniversity of WaterlooWaterlooCanada

Personalised recommendations