Skip to main content
SpringerLink
Log in

Automated Testing and Debugging of SAT and QBF Solvers

  • Conference paper
Book cover Theory and Applications of Satisfiability Testing – SAT 2010 (SAT 2010)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 6175))

Abstract

Robustness and correctness are essential criteria for SAT and QBF solvers. We develop automated testing and debugging techniques designed and optimized for SAT and QBF solver development. Our fuzz testing techniques are able to find critical solver defects that lead to crashes, invalid satisfying assignments and incorrect satisfiability results. Moreover, we show that sequential and concurrent delta debugging techniques are highly effective in minimizing failure-inducing inputs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ansótegui, C., Béjar, R., Fernández, C., Mateu, C.: Generating Hard SAT/CSP Instances Using Expander Graphs. In: AAAI (2008)

    Google Scholar 

  2. Babić, D.: Exploiting Structure for Scalable Software Verification. PhD thesis, University of British Columbia, Vancouver, Canada (2008)

    Google Scholar 

  3. Benedetti, M.: sKizzo: A Suite to Evaluate and Certify QBFs. In: Nieuwenhuis, R. (ed.) CADE 2005. LNCS (LNAI), vol. 3632, pp. 369–376. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  4. Le Berre, D.: SAT4J: Bringing the Power of SAT Technology to the Java Platform, http://www.sat4j.org

  5. Le Berre, D., Roussel, O., Simon, L.: SAT 2009 Competitive Events Booklet - Preliminary Version. In: SAT competition solver descriptions (September 2009), http://www.cril.univ-artois.fr/SAT09/solvers/booklet.pdf

  6. Biere, A.: Resolve and Expand. In: SAT (Selected Papers) (2004)

    Google Scholar 

  7. Biere, A.: PicoSAT Essentials. JSAT 4 (2008)

    Google Scholar 

  8. Biere, A., Heule, M., van Maaren, H., Walsh, T. (eds.): Handbook of Satisfiability. IOS Press, Amsterdam (2009)

    MATH  Google Scholar 

  9. Bofill, M., Nieuwenhuis, R., Oliveras, A., Rodríguez-Carbonell, E., Rubio, A.: The Barcelogic SMT Solver. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 294–298. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  10. Bregman, D., Mitchell, D.: The SAT solver MXC, Version 0.5., SAT competition solver description (2007)

    Google Scholar 

  11. Brummayer, R., Biere, A.: Fuzzing and Delta-Debugging SMT Solvers. In: SMT. ACM International Conference Proceedings Series. ACM, New York (2009)

    Google Scholar 

  12. Kleine Büning, H., Karpinski, M., Flögel, A.: Resolution for Quantified Boolean Formulas. Inf. Comput. 117(1), 12–18 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  13. Cadoli, M., Schaerf, M., Giovanardi, A., Giovanardi, M.: An Algorithm to Evaluate Quantified Boolean Formulae and Its Experimental Evaluation. J. Autom. Reasoning 28 (2002)

    Google Scholar 

  14. Chen, H., Interian, Y.: A Model for Generating Random Quantified Boolean Formulas. In: IJCAI (2005)

    Google Scholar 

  15. Claessen, K., Hughes, J.: QuickCheck: a Lightweight Tool for Random Testing of Haskell Programs. ICFP 35(9) (2000)

    Google Scholar 

  16. Creignou, N., Daudé, H., Egly, U., Rossignol, R.: New Results on the Phase Transition for Random Quantified Boolean Formulas. In: Kleine Büning, H., Zhao, X. (eds.) SAT 2008. LNCS, vol. 4996, pp. 34–47. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  17. Davis, M., Logemann, G., Loveland, D.: A Machine Program for Theorem-Proving. ACM Commun. 5 (1962)

    Google Scholar 

  18. Eén, N., Sörensson, N.: An extensible SAT-solver. In: Giunchiglia, E., Tacchella, A. (eds.) SAT 2003. LNCS, vol. 2919, pp. 502–518. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  19. Freeman, J.W.: Improvements to Propositional Satisfiability Search Algorithms. PhD thesis, Depart. of Comp. and Inf. Science, University of Pennsylvania (1995)

    Google Scholar 

  20. Gebser, M., Kaufmann, B., Schaub, T.: The Conflict-Driven Answer Set Solver clasp: Progress Report. In: Erdem, E., Lin, F., Schaub, T. (eds.) LPNMR 2009. LNCS, vol. 5753, pp. 509–514. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  21. Van Gelder, A.: Extracting (Easily) Checkable Proofs from a Satisfiability Solver that Employs both Preorder and Postorder Resolution. In: AMAI (2002)

    Google Scholar 

  22. Gent, I., Walsh, T.: The SAT Phase Transition. In: ECAI (1994)

    Google Scholar 

  23. Gent, I., Walsh, T.: Beyond NP: the QSAT phase transition. In: AAAI/IAAI (1999)

    Google Scholar 

  24. Giunchiglia, E., Narizzano, M., Tacchella, A.: QuBE++: An Efficient QBF Solver. In: Hu, A.J., Martin, A.K. (eds.) FMCAD 2004. LNCS, vol. 3312, pp. 201–213. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  25. Hamadi, Y., Jabbour, S.: ManySAT: a Parallel SAT Solver. JSAT 6 (2009)

    Google Scholar 

  26. Heule, M.: SmArT Solving: Tools and Techniques for Satisfiability Solvers. PhD thesis, TU Delft (2008)

    Google Scholar 

  27. Horie, S., Watanabe, O.: Hard instance generation for SAT. CoRR, cs.CC/9809117 (1998)

    Google Scholar 

  28. Hsu, E., McIlraith, S.: VARSAT: Integrating Novel Probabilistic Interference Techniques with DPLL Search. In: Kullmann, O. (ed.) SAT 2009. LNCS, vol. 5584, pp. 377–390. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  29. Huang, J.: TINISAT in SAT Competition 2007. SAT competition solver description (2007)

    Google Scholar 

  30. Jain, H., Clarke, E.: SAT Solver Descriptions: CMUSAT-Base and CMUSAT. SAT competition solver description (2007)

    Google Scholar 

  31. Jussila, T., Biere, A., Sinz, C., Kröning, D., Wintersteiger, C.: A First Step Towards a Unified Proof Checker for QBF. In: Marques-Silva, J., Sakallah, K.A. (eds.) SAT 2007. LNCS, vol. 4501, pp. 201–214. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  32. Kern, C., Khaleghi, M., Kugele, S., Schallhart, C., Tautschnig, M., Weis, A.: SAT 7 - Engineering a Modular SAT-Solver. SAT competition solver description (2007)

    Google Scholar 

  33. Letz, R.: Lemma and Model Caching in Decision Procedures for Quantified Boolean Formulas. In: Egly, U., Fermüller, C. (eds.) TABLEAUX 2002. LNCS (LNAI), vol. 2381, p. 160. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  34. Lewis, M., Schubert, T., Becker, B.: Multithreaded SAT Solving. In: Asia and South Pacific DAC (2007)

    Google Scholar 

  35. Lonsing, F.: DepQBF 0.1 Source Code (2010), http://fmv.jku.at/depqbf/

  36. Miller, B., Koski, D., Lee, C., Maganty, V., Murthy, R., Natarajan, A., Steidl, J.: Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services. Technical Report CS-TR-1995-1268, University of Wisconsin, Madison (1995)

    Google Scholar 

  37. Misherghi, G., Su, Z.: HDD: Hierarchical Delta Debugging. In: ICSE, pp. 142–151. ACM, New York (2006)

    Chapter  Google Scholar 

  38. Narizzano, M., Peschiera, C., Pulina, L., Tacchella, A.: Evaluating and Certifying QBFs: A Comparison of State-of-the-Art Tools. AI Commun. 22 (2009)

    Google Scholar 

  39. Pan, G., Vardi, M.: Symbolic Decision Procedures for QBF. In: Wallace, M. (ed.) CP 2004. LNCS, vol. 3258, pp. 453–467. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  40. Pennock, D., Stout, Q.: Exploiting a Theory of Phase Transitions in Three-Satisfiability Problems. In: AAAI/IAAI, vol. 1 (1996)

    Google Scholar 

  41. Pipatsrisawat, K., Darwiche, A.: RSat 2.0: SAT Solver Description. Technical Report D–153, Automated Reasoning Group, CSD, UCLA (2007)

    Google Scholar 

  42. Ranise, S., Tinelli, C.: The SMT-LIB Standard: Version 1.2. Technical report, Department of Computer Science, The University of Iowa (2006)

    Google Scholar 

  43. Samulowitz, H.: MiniQBF Solver (2010), http://miniqbf.spaces.live.com/

  44. Samulowitz, H., Bacchus, F.: Using SAT in QBF. In: van Beek, P. (ed.) CP 2005. LNCS, vol. 3709, pp. 578–592. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  45. Sutton, M., Greene, A., Amini, P.: Fuzzing - Brute Force Vulnerability Discovery. Pearson Ed., London (2007)

    Google Scholar 

  46. Takanen, A., Demott, J., Miller, C.: Fuzzing for Software Security Testing and Quality Assurance. Artech House (2008)

    Google Scholar 

  47. Tseitin, G.: On the Complexity of Proofs in Propositional Logics. Automation of Reasoning: Classical Papers in Computational Logic 1967-1970 2 (1983)

    Google Scholar 

  48. Xu, K., Boussemart, F., Hemery, F., Lecoutre, C.: A Simple Model to Generate Hard Satisfiable Instances. CoRR, abs/cs/0509032 (2005)

    Google Scholar 

  49. Yu, Y., Malik, S.: Validating the Result of a Quantified Boolean Formula (QBF) Solver: Theory and Practice. In: Asia and South Pacific DAC (2005)

    Google Scholar 

  50. Zeller, A.: Why Programs Fail. A Guide to Systematic Debugging. Morgan Kaufmann, San Francisco (2005)

    Google Scholar 

  51. Zeller, A., Hildebrandt, R.: Simplifying and Isolating Failure-Inducing Input. IEEE Transactions on Software Engineering 28(2), 183–200 (2002)

    Article  Google Scholar 

  52. Zhang, L., Malik, S.: Validating SAT Solvers Using an Independent Resolution-Based Checker: Practical Implementations and Other Applications. In: DATE 2003 (2003)

    Google Scholar 

  53. Zhang, L., Malik, S.: Towards Symmetric Treatment of Conflicts And Satisfaction in Quantified Boolean Satisfiability Solver. In: Van Hentenryck, P. (ed.) CP 2002. LNCS, vol. 2470, p. 200. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Formal Models and Verification, Johannes Kepler University, Linz, Austria

    Robert Brummayer, Florian Lonsing & Armin Biere

Authors
  1. Robert Brummayer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Florian Lonsing
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Armin Biere
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Technion, Technion City, 32000, Haifa, Israel

    Ofer Strichman

  2. Vienna University of Technology, Favoritenstr. 9-11, 1040, Vienna, Austria

    Stefan Szeider

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Brummayer, R., Lonsing, F., Biere, A. (2010). Automated Testing and Debugging of SAT and QBF Solvers. In: Strichman, O., Szeider, S. (eds) Theory and Applications of Satisfiability Testing – SAT 2010. SAT 2010. Lecture Notes in Computer Science, vol 6175. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-14186-7_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-14186-7_6

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-14185-0

  • Online ISBN: 978-3-642-14186-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation