The SAT2002 competition

  • Laurent Simon
  • Daniel Le Berre
  • Edward A. Hirsch
Article

Abstract

SAT Competition 2002 held in March–May 2002 in conjunction with SAT 2002 (the Fifth International Symposium on the Theory and Applications of Satisfiability Testing). About 30 solvers and 2300 benchmarks took part in the competition, which required more than 2 CPU years to complete the evaluation. In this report, we give the results of the competition, try to interpret them, and give suggestions for future competitions.

Keywords

Boolean satisfiability (SAT) empirical evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    P.A. Abdulla, P. Bjesse and N. Eén, Symbolic reachability analysis based on SAT-solvers, in: Proceedings of the 6th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’2000) (2000).Google Scholar
  2. [2]
    F. Bacchus, Enhancing Davis Putnam with extended binary clause reasoning, in: Proceedings of National Conference on Artificial Intelligence (AAAI-2002) (2002).Google Scholar
  3. [3]
    F. Bacchus, Exploring the computational tradeoff of more reasoning and less searching, in: [49, pp. 7–16] (2002).Google Scholar
  4. [4]
    L. Baptista and J.P. Marques-Silva, Using randomization and learning to solve hard real-world instances of satisfiability, in: Proceedings of the 6th International Conference on Principles and Practice of Constraint Programming (CP) (2000).Google Scholar
  5. [5]
    R.J.J. Bayardo and R.C. Schrag, Using CSP look-back techniques to solve real-world SAT instances, in: Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI’97) (AMS, Providence, RI, 1997) pp. 203–208.Google Scholar
  6. [6]
    A. Biere, A. Cimatti, E.M. Clarke, M. Fujita and Y. Zhu, Symbolic model checking using SAT procedures instead of BDDs, in: Proceedings of Design Automation Conference (DAC’99) (1999).Google Scholar
  7. [7]
    M. Buro and H.K. Büning, Report on a SAT competition, Bulletin of the European Association for Theoretical Computer Science 49 (1993) 143–151.MATHGoogle Scholar
  8. [8]
    C.-M. Li, B. Jurkowiak and P.W. Purdom Jr, Integrating symmetry breaking into a DLL procedure, in: [49, pp. 149–155] (2002).Google Scholar
  9. [9]
    A.E. Caldwell, A.B. Kahng and I.L. Markov, Toward CAD-IP reuse: The MARCO GSRC bookshelf of fundamental CAD algorithms, IEEE Design and Test (May 2002) 72-81.Google Scholar
  10. [10]
    P. Chatalic and L. Simon, Multi-resolution on compressed sets of clauses, in: Twelth International Conference on Tools with Artificial Intelligence (ICTAI’00) (2000) pp. 2–10.Google Scholar
  11. [11]
    S.A. Cook, The complexity of theorem-proving procedures, in: Proceedings of the Third IEEE Symposium on the Foundations of Computer Science (1971) pp. 151–158.Google Scholar
  12. [12]
    F. Copty, L. Fix, E. Giunchiglia, G. Kamhi, A. Tacchella and M. Vardi, Benefits of bounded model checking at an industrial setting, in: Proc. of CAV (2001).Google Scholar
  13. [13]
    E. Dantsin, A. Goerdt, E.A. Hirsch, R. Kannan, J. Kleinberg, C. Papadimitriou, P. Raghavan and U. Schöning, Deterministic (2 − 2 (k + 1 )n algorithm for k-SAT based on local search, Theoretical Computer Science 189(1) (2002) 69–83.CrossRefGoogle Scholar
  14. [14]
    M. Davis, G. Logemann and D. Loveland, A machine program for theorem proving, Communications of the ACM 5(7) (1962) 394–397.CrossRefMATHMathSciNetGoogle Scholar
  15. [15]
    M. Davis and H. Putnam, A computing procedure for quantification theory, Journal of the ACM 7(3) (1960) 201–215.CrossRefMATHMathSciNetGoogle Scholar
  16. [16]
    O. Dubois, P. André, Y. Boufkhad and J. Carlier, SAT versus UNSAT, in: [29, pp. 415–436] (1996).Google Scholar
  17. [17]
    O. Dubois and G. Dequen, A backbone-search heuristic for efficient solving of hard 3-SAT formulae, in: Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI’01), Seattle, WA (2001).Google Scholar
  18. [18]
    M.D. Ernst, T.D. Millstein and D.S. Weld, Automatic SAT-compilation of planning problems, in: [28, pp. 1169–1176] (1997).Google Scholar
  19. [19]
    F. Aloul, A. Ramani, I. Markov and K. Sakallah, Solving difficult SAT instances in the presence of symmetry, in: Design Automation Conference (DAC), New Orleans, LO (2002) pp. 731–736.Google Scholar
  20. [20]
    J.W. Freeman, Improvements to propositional satisfiability search algorithms, Ph.D. thesis, Departement of Computer and Information Science, University of Pennsylvania, Philadelphia, PA (1995).Google Scholar
  21. [21]
    E. Goldberg and Y. Novikov, BerkMin: A fast and robust SAT-solver, in: Design, Automation, and Test in Europe (DATE ‘02) (2002) pp. 142–149.Google Scholar
  22. [22]
    C.P. Gomes, B. Selman and H. Kautz, Boosting combinatorial search through randomization, in: Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI’98), Madison, WI (1998) pp. 431–437.Google Scholar
  23. [23]
    E.A. Hirsch, SAT local search algorithms: Worst-case study, Journal of Automated Reasoning 24(1/2) (2000) 127–143. Also reprinted in Highlights of Satisfiability Research in the Year 2000, Frontiers in Artificial Intelligence and Applications, Vol. 63 (IOS Press, 2000).CrossRefMATHMathSciNetGoogle Scholar
  24. [24]
    E.A. Hirsch, New worst-case upper bounds for SAT, Journal of Automated Reasoning 24(4) (2000) 397–420. Also reprinted in Highlights of Satisfiability Research in the Year 2000, Frontiers in Artificial Intelligence and Applications, Vol. 63 (IOS Press, 2000).CrossRefMATHMathSciNetGoogle Scholar
  25. [25]
    E.A. Hirsch and A. Kojevnikov, UnitWalk: A new SAT solver that uses local search guided by unit clause elimination, Annals of Mathematics and Artificial Intelligence 43 (2005) 91–111.CrossRefMATHMathSciNetGoogle Scholar
  26. [26]
    J.N. Hooker, Needed: An empirical science of algorithms, Operations Research 42(2) (1994) 201–212.CrossRefMATHGoogle Scholar
  27. [27]
    J.N. Hooker, Testing heuristics: We have it all wrong, Journal of Heuristics (1996) 32–42.Google Scholar
  28. [28]
    IJCAI97, Proceedings of the 15th International Joint Conference on Artificial Intelligence (IJCAI’97), Nagoya, Japan (1997).Google Scholar
  29. [29]
    D. Johnson and M. Trick (eds.), Second DIMACS Implementation Challenge: Cliques, Coloring and Satisfiability, DIMACS Series in Discrete Mathematics and Theoretical Computer Science, Vol. 26 (American Mathematical Society, 1996).Google Scholar
  30. [30]
    H. Kautz and B. Selman (eds.), Proceedings of the Workshop on Theory and Applications of Satisfiability Testing (SAT2001), LICS 2001 Workshop on Theory and Applications of Satisfiability Testing (SAT 2001) (Elsevier Science, 2001).Google Scholar
  31. [31]
    H.A. Kautz and B. Selman, Planning as satisfiability, in: Proceedings of the 10th European Conference on Artificial Intelligence (ECAI’92) (1992) pp. 359–363.Google Scholar
  32. [32]
    H.A. Kautz and B. Selman, Pushing the envelope: Planning, propositional logic, and stochastic search, in: Proceedings of the 12th National Conference on Artificial Intelligence (AAAI’96) (1996) pp. 1194–1201.Google Scholar
  33. [33]
    E. Koutsoupias and C.H. Papadimitriou, On the greedy algorithm for satisfiability, Information Processing Letters 43(1) (1992) 53–55.CrossRefMATHMathSciNetGoogle Scholar
  34. [34]
    O. Kullmann, First report on an adaptive density based branching rule for DLL-like SAT solvers, using a database for mixed random conjunctive normal forms created using the Advanced Encryption Standard (AES), Technical Report CSR 19-2002, University of Wales Swansea, Computer Science Report Series (2002). (Extended version of [36].)Google Scholar
  35. [35]
    O. Kullmann, Investigating the behaviour of a SAT solver on random formulas, Annals of Mathematics and Artificial Intelligence (2002).Google Scholar
  36. [36]
    O. Kullmann, Towards an adaptive density based branching rule for SAT solvers, using a database for mixed random conjunctive normal forms built upon the Advanced Encryption Standard (AES), in: [49] (2002).Google Scholar
  37. [37]
    C.-M. Li, A constrained based approach to narrow search trees for satisfiability, Information Processing Letters 71 (1999) 75–80.CrossRefMATHMathSciNetGoogle Scholar
  38. [38]
    C.-M. Li, Integrating equivalency reasoning into Davis-Putnam procedure, in: Proceedings of the 17th National Conference in Artificial Intelligence (AAAI’00), Austin, TX (2000) pp. 291–296.Google Scholar
  39. [39]
    C.-M. Li and Anbulagan, Heuristics based on unit propagation for satisfiability problems, in: [28, pp. 366–371] (1997).Google Scholar
  40. [40]
    I. Lynce and J.P. Marques Silva, Efficient data structures for backtrack search SAT solvers, in: [49] (2002).Google Scholar
  41. [41]
    I. Lynce, L. Baptista and J.P. Marques Silva, Stochastic systematic search algorithms for satisfiability, in: [30] (2001).Google Scholar
  42. [42]
    J.P. Marques-Silva and K.A. Sakallah, GRASP — A new search algorithm for satisfiability, in: Proceedings of IEEE/ACM International Conference on Computer-Aided Design (1996) pp. 220–227.Google Scholar
  43. [43]
    M.W. Moskewicz, C.F. Madigan, Y. Zhao, L. Zhang and S. Malik, Chaff: Engineering an efficient SAT solver, in: Proceedings of the 38th Design Automation Conference (DAC’01) (2001) pp. 530–535.Google Scholar
  44. [44]
    F. Okushi and A. Van Gelder, Persistent and quasi-persistent lemmas in propositional model elimination, in: (Electronic) Proc. 6th Int’l Symposium on Artificial Intelligence and Mathematics (2000).; Annals of Mathematics and Artificial Intelligence 40(3–4) (2004) 373–402.CrossRefMATHMathSciNetGoogle Scholar
  45. [45]
    R. Ostrowski, E. Grégoire, B. Mazure and L. Sais, Recovering and exploiting structural knowledge from CNF formulas, in: Proc. of the Eighth International Conference on Principles and Practice of Constraint Programming (CP’2002), Ithaca, NY (2002).Google Scholar
  46. [46]
    R. Paturi, P. Pudlàk and F. Zane, Satisfiability coding lemma, in: Proceedings of the 38th Annual IEEE Symposium on Foundations of Computer Science, FOCS’97 (1997) pp. 566–574.Google Scholar
  47. [47]
    S. Prestwich, A SAT approach to query optimization in mediator systems, in: [49, pp. 252-259] (2002).Google Scholar
  48. [48]
    S.D. Prestwich, Randomised backtracking for linear pseudo-Boolean constraint problems, in: Proceedings of Fourth International Workshop on Integration of AI and OR techniques in Constraint Programming for Combinatorial Optimisation Problems (2002).Google Scholar
  49. [49]
    SAT2002, Fifth International Symposium on the Theory and Applications of Satisfiability Testing, Cincinnati, OH (2002).Google Scholar
  50. [50]
    R. Schuler, U. Schöning, O. Watanabe and T. Hofmeister, A probabilistic 3-SAT algorithm further improved, in: Proceedings of 19th International Symposium on Theoretical Aspects of Computer Science, STACS 2002 (2002).Google Scholar
  51. [51]
    B. Selman, H.A. Kautz and B. Cohen, Noise strategies for improving local search, in: Proceedings of the 12th National Conference on Artificial Intelligence (AAAI’94), Seattle (1994) pp. 337–343.Google Scholar
  52. [52]
    B. Selman, H. Levesque and D. Mitchell, A new method for solving hard satisfiability problems, in: Proceedings of the 10th National Conference on Artificial Intelligence (AAAI’92) (1992) pp. 440–446.Google Scholar
  53. [53]
    Y. Shang and B.W. Wah, A discrete Lagrangian-based global-search method for solving satisfiability problems, Journal of Global Optimization 12(1) (1998) 61–99.CrossRefMATHMathSciNetGoogle Scholar
  54. [54]
    L. Simon and P. Chatalic, SATEx: a Web-based framework for SAT experimentation, in: [30] (2001); http://www.lri.fr/~simon/satex.Google Scholar
  55. [55]
    G. Sutcliff and C. Suttner, Evaluating general purpose automated theorem proving systems, Artificial Intelligence 131 (2001) 39–54.CrossRefMathSciNetGoogle Scholar
  56. [56]
    G.S. Tseitin, On the complexity of derivation in the propositional calculus, in: Structures in Constructive Mathematics and Mathematical Logic, Part II, ed. A.O. Slisenko (Consultants Bureau, New York, 1970) 115–125. Translated from Russian.CrossRefGoogle Scholar
  57. [57]
    A. Urquhart, Hard examples for resolution, Journal of the Association for Computing Machinery 34(1) (1987) 209–219.CrossRefMATHMathSciNetGoogle Scholar
  58. [58]
    A. Van Gelder, Autarky pruning in propositional model elimination reduces failure redundancy, Journal of Automated Reasoning 23(2) (1999) 137–193.CrossRefMATHMathSciNetGoogle Scholar
  59. [59]
    A. Van Gelder, Extracting (easily) checkable proofs from a satisfiability solver that employs both pre-order and postorder resolution, in: Seventh Int’l Symposium on AI and Mathematics, Fort Lauderdale, FL (2002).Google Scholar
  60. [60]
    A. Van Gelder, Generalizations of watched literals for backtracking search, in: Seventh Int’l Symposium on AI and Mathematics, Fort Lauderdale, FL (2002).Google Scholar
  61. [61]
    A. Van Gelder and F. Okushi, Lemma and Cut strategies for propositional model elimination, Annals of Mathematics and Artificial Intelligence 26(1–4) (1999) 113–132.CrossRefMATHMathSciNetGoogle Scholar
  62. [62]
    A. Van Gelder and Y.K. Tsuji, Satisfiability testing with more reasoning and less guessing, in: [29, pp. 559–586] (1996).Google Scholar
  63. [63]
    M. Velev and R. Bryant, Effective use of Boolean satisfiability procedures in the formal verification of superscalar and VLIW microprocessors, in: Proceedings of the 38th Design Automation Conference (DAC ‘01) (2001) pp. 226–231.Google Scholar
  64. [64]
    J. Warners and H. van Maaren, Solving satisfiability problems using elliptic approximations: Effective branching rules, Discrete Applied Mathematics 107 (2000) 241–259.CrossRefMATHMathSciNetGoogle Scholar
  65. [65]
    H. Zhang, SATO: An efficient propositional prover, in: Proceedings of the International Conference on Automated Deduction (CADE’97), Lecture Notes in Artificial Intelligence, Vol. 1249 (1997) pp. 272-275.Google Scholar
  66. [66]
    H. Zhang and M.E. Stickel, An efficient algorithm for unit propagation, in: Proceedings of the Fourth International Symposium on Artificial Intelligence and Mathematics (AI-MATH’96), Fort Lauderdale, FL (1996).Google Scholar
  67. [67]
    L. Zhang, C.F. Madigan, M.W. Moskewicz and S. Malik, Efficient conflict driven learning in a Boolean satisfiability solver, in: International Conference on Computer-Aided Design (ICCAD’01) (2001) pp. 279–285.Google Scholar
  68. [68]
    L. Zheng and P.J. Stuckey, Improving SAT using 2SAT, in: Proceedings of the Twenty-Fifth Australasian Computer Science Conference (ACSC2002), ed. M.J. Oudshoorn, Melbourne, Australia (2002).Google Scholar

Copyright information

© Springer 2004

Authors and Affiliations

  • Laurent Simon
    • 1
  • Daniel Le Berre
    • 2
  • Edward A. Hirsch
    • 3
  1. 1.LRI, U.M.R. CNRS 8623Université Paris-SudOrsayFrance
  2. 2.CRIL, F.R.E. CNRS 2499, Faculté Jean PerrinUniversité d’ ArtoisLensFrance
  3. 3.Steklov Institute of Mathematics at St. PetersburgSt. PetersburgRussia

Personalised recommendations