Skip to main content
SpringerLink
Log in

Evaluating Instance Generators by Configuration

  • Conference paper
  • First Online:
Learning and Intelligent Optimization (LION 2014)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 8426))

Included in the following conference series:

Abstract

The propositional satisfiability problem (SAT) is one of the most prominent and widely studied NP-hard problems. The development of SAT solvers, whether it is carried out manually or through the use of automated design tools such as algorithm configurators, depends substantially on the sets of benchmark instances used for performance evaluation. Since the supply of instances from real-world applications of SAT is limited, and artificial instance distributions such as Uniform Random \(k\)-SAT are known to have markedly different structure, there has been a long-standing interest in instance generators capable of producing ‘realistic’ SAT instances that could be used during development as proxies for real-world instances. However, it is not obvious how to assess the quality of the instances produced by any such generator. We propose a new approach for evaluating the usefulness of an arbitrary set of instances for use as proxies during solver development, and introduce a new metric, \(Q\)-score, to quantify this. We apply our approach on several artificially generated and real-world benchmark sets and quantitatively compare their usefulness for developing competitive SAT solvers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Unfortunately, this instance generator is not publicly available.

  2. 2.

    ParamILS can only configure over finite, discretized configuration spaces. Parameters taking arbitrary integers or real numbers were manually discretized to a small number of representative values (\(<10\)), from which the spaces above were computed.

References

  1. Achlioptas, D., Gomes, C., Kautz, H., Selman, B.: Generating satisfiable problem instances. In: Proceedings of the National Conference on Artificial Intelligence, pp. 256–261. Menlo Park, CA, Cambridge, MA, London, AAAI Press, MIT Press, 1999 (2000)

    Google Scholar 

  2. Ansótegui, C., Bonet, M.L., Levy, J.: Towards industrial-like random SAT instances. In: Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), pp. 387–392 (2009)

    Google Scholar 

  3. Ansótegui, C., Bonet, M.L., Levy, J., Manyà, F.: Measuring the hardness of SAT instances. In: Proceedings of the Twenty-Third National Conference on Artificial Intelligence (AAAI-08), pp. 222–229 (2008)

    Google Scholar 

  4. Babić, D., Hu, A.J.: Structural abstraction of software verification conditions. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 366–378. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  5. Babić, D., Hutter, F.: Spear theorem prover. Solver Description, SAT Race 2008 (2008)

    Google Scholar 

  6. Armin, B.: Lingeling, Plingeling PicoSAT and PrecoSAT at SAT race. Technical report 10/1, FMV Report Series, Institute for Formal Models and Verification, Johannes Kepler University (2010)

    Google Scholar 

  7. Brummayer, R., Lonsing, F., Biere, A.: Automated testing and debugging of SAT and QBF solvers. In: Strichman, O., Szeider, S. (eds.) SAT 2010. LNCS, vol. 6175, pp. 44–57. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  8. Burg, S., Kottler, S., Kaufmann, M.: Creating industrial-like SAT instances by clustering and reconstruction. In: Cimatti, A., Sebastiani, R. (eds.) SAT 2012. LNCS, vol. 7317, pp. 471–472. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  9. Chvátal, V., Szemerédi, E.: Many hard examples for resolution. J. ACM 35(4), 759–768 (1988)

    Article  MATH  Google Scholar 

  10. Dequen, G., Dubois, O.: An efficient approach to solving random \(k\)-SAT problems. J. Autom. Reason. 37(4), 261–276 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  11. Gomes, C.P., Selman, B., et al.: Problem structure in the presence of perturbations. In: Proceedings of the National Conference on Artificial Intelligence, pp. 221–226. Wiley (1997)

    Google Scholar 

  12. Haanpää, H., Järvisalo, M., Kaski, P., Niemelä, I.: Hard satisfiable clause sets for benchmarking equivalence reasoning techniques. J. Satisf. Boolean Model. Comput. 2(1–4), 27–46 (2006)

    MATH  Google Scholar 

  13. Heule, M.J.H., van Maaren, H.: Whose side are you on? finding solutions in a biased search-tree. J. Satisf. Boolean Model. Comput. 4, 117–148 (2008)

    MATH  Google Scholar 

  14. Hoos, H.H.: Programming by optimization. Commun. ACM 55, 70–80 (2011)

    Article  Google Scholar 

  15. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 5. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  16. Hutter, F., Babić, D., Hoos, H.H., Hu, A.J.: Boosting verification by automatic tuning of decision procedures. In: Proceedings of the Seventh International Conference on Formal Methods in Computer-Aided Design (FMCAD-07), pp. 27–34 (2007)

    Google Scholar 

  17. Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intell. Res. 36, 267–306 (2009)

    MATH  Google Scholar 

  18. Hutter, F., Hoos, H.H., Stützle, T.: Automatic algorithm configuration based on local search. In: Proceedings of the Twenty-Second National Conference on Artificial Intelligence (AAAI-07), pp. 1152–1157 (2007)

    Google Scholar 

  19. Järvisalo, M., Kaski, P., Koivisto, M., Korhonen, J.H.: Finding efficient circuits for ensemble computation. In: Cimatti, A., Sebastiani, R. (eds.) SAT 2012. LNCS, vol. 7317, pp. 369–382. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  20. Kautz, H., Selman, B.: Ten challenges Redux: recent progress in propositional reasoning and search. In: Rossi, F. (ed.) CP 2003. LNCS, vol. 2833, pp. 1–18. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  21. KhudaBukhsh, A.R.: SATenstein: automatically building local search SAT solvers from components. Master’s thesis, University of British Columbia (2009)

    Google Scholar 

  22. KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K.: SATenstein: automatically building local search SAT solvers from components. In: Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), pp. 517–524 (2009)

    Google Scholar 

  23. Li, C.M., Anbulagan, : Look-ahead versus look-back for satisfiability problems. In: Smolka, G. (ed.) CP 1997. LNCS, vol. 1330, pp. 341–355. Springer, Heidelberg (1997)

    Chapter  Google Scholar 

  24. Rish, I., Dechter, R.: Resolution versus search: two strategies for sat. J. Autom. Reason. 24(1), 225–275 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  25. Selman, B., Kautz, H., McAllester, D.: Ten challenges in propositional reasoning and search. In: Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI-97), pp. 50–54 (1997)

    Google Scholar 

  26. Slater, A.: Modelling more realistic SAT problems. In: McKay, B., Slaney, J.K. (eds.) AI 2002. LNCS (LNAI), vol. 2557, pp. 591–602. Springer, Heidelberg (2002)

    Google Scholar 

  27. Sörensson, N., Eén, N.: Minisat v1.13 - a SAT solver with conflict-clause minimization. In: Poster, Eighth International Conference on Theory and Applications of Satisfiability Testing (SAT-05) (2005)

    Google Scholar 

  28. Tompkins, D.A.D., Balint, A., Hoos, H.H.: Captain Jack: new variable selection heuristics in local search for SAT. In: Sakallah, K.A., Simon, L. (eds.) SAT 2011. LNCS, vol. 6695, pp. 302–316. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  29. Tompkins, D.A.D., Hoos, H.H.: Dynamic scoring functions with variable expressions: new SLS methods for solving SAT. In: Strichman, O., Szeider, S. (eds.) SAT 2010. LNCS, vol. 6175, pp. 278–292. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  30. Zarpas, E.: Benchmarking SAT solvers for bounded model checking. In: Bacchus, F., Walsh, T. (eds.) SAT 2005. LNCS, vol. 3569, pp. 340–354. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

Download references

Acknowledgments

This research has been enabled by the use of computing resources provided by WestGrid and Compute/Calcul Canada, and funding provided by the NSERC Canada Graduate Scholarships and Discovery Grants Programs.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of British Columbia, Vancouver, Canada

    Sam Bayless & Holger H. Hoos

  2. David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada

    Dave A. D. Tompkins

Authors
  1. Sam Bayless
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dave A. D. Tompkins
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Holger H. Hoos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sam Bayless .

Editor information

Editors and Affiliations

  1. Department of Industrial and Systems Engineering, University of Florida, Gainesville, Florida, USA

    Panos M. Pardalos

  2. AT&T Labs Research, Middletown, New Jersey, USA

    Mauricio G.C. Resende

  3. University of Florida, Gainesville, Florida, USA

    Chrysafis Vogiatzis

  4. Department of Industrial and Systems Engineering, University of Florida, Gainesville, Florida, USA

    Jose L. Walteros

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Bayless, S., Tompkins, D.A.D., Hoos, H.H. (2014). Evaluating Instance Generators by Configuration. In: Pardalos, P., Resende, M., Vogiatzis, C., Walteros, J. (eds) Learning and Intelligent Optimization. LION 2014. Lecture Notes in Computer Science(), vol 8426. Springer, Cham. https://doi.org/10.1007/978-3-319-09584-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-09584-4_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-09583-7

  • Online ISBN: 978-3-319-09584-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation