# The SAT2002 competition

Article

First Online:

- 81 Downloads
- 10 Citations

## Abstract

SAT Competition 2002 held in March–May 2002 in conjunction with SAT 2002 (the Fifth International Symposium on the Theory and Applications of Satisfiability Testing). About 30 solvers and 2300 benchmarks took part in the competition, which required more than 2 CPU years to complete the evaluation. In this report, we give the results of the competition, try to interpret them, and give suggestions for future competitions.

## Keywords

Boolean satisfiability (SAT) empirical evaluation## Preview

Unable to display preview. Download preview PDF.

## References

- [1]P.A. Abdulla, P. Bjesse and N. Eén, Symbolic reachability analysis based on SAT-solvers, in:
*Proceedings of the 6th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’2000)*(2000).Google Scholar - [2]F. Bacchus, Enhancing Davis Putnam with extended binary clause reasoning, in:
*Proceedings of National Conference on Artificial Intelligence (AAAI-2002)*(2002).Google Scholar - [3]F. Bacchus, Exploring the computational tradeoff of more reasoning and less searching, in: [49, pp. 7–16] (2002).Google Scholar
- [4]L. Baptista and J.P. Marques-Silva, Using randomization and learning to solve hard real-world instances of satisfiability, in:
*Proceedings of the 6th International Conference on Principles and Practice of Constraint Programming (CP)*(2000).Google Scholar - [5]R.J.J. Bayardo and R.C. Schrag, Using CSP look-back techniques to solve real-world SAT instances, in:
*Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI’97)*(AMS, Providence, RI, 1997) pp. 203–208.Google Scholar - [6]A. Biere, A. Cimatti, E.M. Clarke, M. Fujita and Y. Zhu, Symbolic model checking using SAT procedures instead of BDDs, in:
*Proceedings of Design Automation Conference (DAC’99)*(1999).Google Scholar - [7]M. Buro and H.K. Büning, Report on a SAT competition, Bulletin of the European Association for Theoretical Computer Science 49 (1993) 143–151.zbMATHGoogle Scholar
- [8]C.-M. Li, B. Jurkowiak and P.W. Purdom Jr, Integrating symmetry breaking into a DLL procedure, in: [49, pp. 149–155] (2002).Google Scholar
- [9]A.E. Caldwell, A.B. Kahng and I.L. Markov, Toward CAD-IP reuse: The MARCO GSRC bookshelf of fundamental CAD algorithms, IEEE Design and Test (May 2002) 72-81.Google Scholar
- [10]P. Chatalic and L. Simon, Multi-resolution on compressed sets of clauses, in:
*Twelth International Conference on Tools with Artificial Intelligence (ICTAI’00)*(2000) pp. 2–10.Google Scholar - [11]S.A. Cook, The complexity of theorem-proving procedures, in:
*Proceedings of the Third IEEE Symposium on the Foundations of Computer Science*(1971) pp. 151–158.Google Scholar - [12]F. Copty, L. Fix, E. Giunchiglia, G. Kamhi, A. Tacchella and M. Vardi, Benefits of bounded model checking at an industrial setting, in:
*Proc. of CAV*(2001).Google Scholar - [13]E. Dantsin, A. Goerdt, E.A. Hirsch, R. Kannan, J. Kleinberg, C. Papadimitriou, P. Raghavan and U. Schöning, Deterministic (2 − 2 (
*k*+ 1 )^{n}algorithm for*k*-SAT based on local search, Theoretical Computer Science 189(1) (2002) 69–83.CrossRefGoogle Scholar - [14]M. Davis, G. Logemann and D. Loveland, A machine program for theorem proving, Communications of the ACM 5(7) (1962) 394–397.CrossRefzbMATHMathSciNetGoogle Scholar
- [15]M. Davis and H. Putnam, A computing procedure for quantification theory, Journal of the ACM 7(3) (1960) 201–215.CrossRefzbMATHMathSciNetGoogle Scholar
- [16]O. Dubois, P. André, Y. Boufkhad and J. Carlier, SAT versus UNSAT, in: [29, pp. 415–436] (1996).Google Scholar
- [17]O. Dubois and G. Dequen, A backbone-search heuristic for efficient solving of hard 3-SAT formulae, in:
*Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI’01),*Seattle, WA (2001).Google Scholar - [18]M.D. Ernst, T.D. Millstein and D.S. Weld, Automatic SAT-compilation of planning problems, in: [28, pp. 1169–1176] (1997).Google Scholar
- [19]F. Aloul, A. Ramani, I. Markov and K. Sakallah, Solving difficult SAT instances in the presence of symmetry, in:
*Design Automation Conference (DAC)*, New Orleans, LO (2002) pp. 731–736.Google Scholar - [20]J.W. Freeman, Improvements to propositional satisfiability search algorithms, Ph.D. thesis, Departement of Computer and Information Science, University of Pennsylvania, Philadelphia, PA (1995).Google Scholar
- [21]E. Goldberg and Y. Novikov, BerkMin: A fast and robust SAT-solver, in:
*Design, Automation, and Test in Europe (DATE ‘02)*(2002) pp. 142–149.Google Scholar - [22]C.P. Gomes, B. Selman and H. Kautz, Boosting combinatorial search through randomization, in:
*Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI’98)*, Madison, WI (1998) pp. 431–437.Google Scholar - [23]E.A. Hirsch, SAT local search algorithms: Worst-case study, Journal of Automated Reasoning 24(1/2) (2000) 127–143. Also reprinted in
*Highlights of Satisfiability Research in the Year 2000*, Frontiers in Artificial Intelligence and Applications, Vol. 63 (IOS Press, 2000).CrossRefzbMATHMathSciNetGoogle Scholar - [24]E.A. Hirsch, New worst-case upper bounds for SAT, Journal of Automated Reasoning 24(4) (2000) 397–420. Also reprinted in
*Highlights of Satisfiability Research in the Year 2000*, Frontiers in Artificial Intelligence and Applications, Vol. 63 (IOS Press, 2000).CrossRefzbMATHMathSciNetGoogle Scholar - [25]E.A. Hirsch and A. Kojevnikov, UnitWalk: A new SAT solver that uses local search guided by unit clause elimination, Annals of Mathematics and Artificial Intelligence 43 (2005) 91–111.CrossRefzbMATHMathSciNetGoogle Scholar
- [26]J.N. Hooker, Needed: An empirical science of algorithms, Operations Research 42(2) (1994) 201–212.CrossRefzbMATHGoogle Scholar
- [27]J.N. Hooker, Testing heuristics: We have it all wrong, Journal of Heuristics (1996) 32–42.Google Scholar
- [28]IJCAI97,
*Proceedings of the 15th International Joint Conference on Artificial Intelligence (IJCAI’97)*, Nagoya, Japan (1997).Google Scholar - [29]D. Johnson and M. Trick (eds.),
*Second DIMACS Implementation Challenge: Cliques, Coloring and Satisfiability*, DIMACS Series in Discrete Mathematics and Theoretical Computer Science, Vol. 26 (American Mathematical Society, 1996).Google Scholar - [30]H. Kautz and B. Selman (eds.),
*Proceedings of the Workshop on Theory and Applications of Satisfiability Testing (SAT2001), LICS 2001 Workshop on Theory and Applications of Satisfiability Testing (SAT 2001)*(Elsevier Science, 2001).Google Scholar - [31]H.A. Kautz and B. Selman, Planning as satisfiability, in:
*Proceedings of the 10th European Conference on Artificial Intelligence*(ECAI’92) (1992) pp. 359–363.Google Scholar - [32]H.A. Kautz and B. Selman, Pushing the envelope: Planning, propositional logic, and stochastic search, in:
*Proceedings of the 12th National Conference on Artificial Intelligence (AAAI’96)*(1996) pp. 1194–1201.Google Scholar - [33]E. Koutsoupias and C.H. Papadimitriou, On the greedy algorithm for satisfiability, Information Processing Letters 43(1) (1992) 53–55.CrossRefzbMATHMathSciNetGoogle Scholar
- [34]O. Kullmann, First report on an adaptive density based branching rule for DLL-like SAT solvers, using a database for mixed random conjunctive normal forms created using the Advanced Encryption Standard (AES), Technical Report CSR 19-2002, University of Wales Swansea, Computer Science Report Series (2002). (Extended version of [36].)Google Scholar
- [35]O. Kullmann, Investigating the behaviour of a SAT solver on random formulas, Annals of Mathematics and Artificial Intelligence (2002).Google Scholar
- [36]O. Kullmann, Towards an adaptive density based branching rule for SAT solvers, using a database for mixed random conjunctive normal forms built upon the Advanced Encryption Standard (AES), in: [49] (2002).Google Scholar
- [37]C.-M. Li, A constrained based approach to narrow search trees for satisfiability, Information Processing Letters 71 (1999) 75–80.CrossRefzbMATHMathSciNetGoogle Scholar
- [38]C.-M. Li, Integrating equivalency reasoning into Davis-Putnam procedure, in:
*Proceedings of the 17th National Conference in Artificial Intelligence (AAAI’00)*, Austin, TX (2000) pp. 291–296.Google Scholar - [39]C.-M. Li and Anbulagan, Heuristics based on unit propagation for satisfiability problems, in: [28, pp. 366–371] (1997).Google Scholar
- [40]I. Lynce and J.P. Marques Silva, Efficient data structures for backtrack search SAT solvers, in: [49] (2002).Google Scholar
- [41]I. Lynce, L. Baptista and J.P. Marques Silva, Stochastic systematic search algorithms for satisfiability, in: [30] (2001).Google Scholar
- [42]J.P. Marques-Silva and K.A. Sakallah, GRASP — A new search algorithm for satisfiability, in:
*Proceedings of IEEE/ACM International Conference on Computer-Aided Design*(1996) pp. 220–227.Google Scholar - [43]M.W. Moskewicz, C.F. Madigan, Y. Zhao, L. Zhang and S. Malik, Chaff: Engineering an efficient SAT solver, in:
*Proceedings of the 38th Design Automation Conference (DAC’01)*(2001) pp. 530–535.Google Scholar - [44]F. Okushi and A. Van Gelder, Persistent and quasi-persistent lemmas in propositional model elimination, in:
*(Electronic) Proc. 6th Int’l Symposium on Artificial Intelligence and Mathematics*(2000).; Annals of Mathematics and Artificial Intelligence 40(3–4) (2004) 373–402.CrossRefzbMATHMathSciNetGoogle Scholar - [45]R. Ostrowski, E. Grégoire, B. Mazure and L. Sais, Recovering and exploiting structural knowledge from CNF formulas, in:
*Proc. of the Eighth International Conference on Principles and Practice of Constraint Programming (CP’2002)*, Ithaca, NY (2002).Google Scholar - [46]R. Paturi, P. Pudlàk and F. Zane, Satisfiability coding lemma, in:
*Proceedings of the 38th Annual IEEE Symposium on Foundations of Computer Science, FOCS’97*(1997) pp. 566–574.Google Scholar - [47]S. Prestwich, A SAT approach to query optimization in mediator systems, in: [49, pp. 252-259] (2002).Google Scholar
- [48]S.D. Prestwich, Randomised backtracking for linear pseudo-Boolean constraint problems, in:
*Proceedings of Fourth International Workshop on Integration of AI and OR techniques in Constraint Programming for Combinatorial Optimisation Problems*(2002).Google Scholar - [49]SAT2002,
*Fifth International Symposium on the Theory and Applications of Satisfiability Testing*, Cincinnati, OH (2002).Google Scholar - [50]R. Schuler, U. Schöning, O. Watanabe and T. Hofmeister, A probabilistic 3-SAT algorithm further improved, in:
*Proceedings of 19th International Symposium on Theoretical Aspects of Computer Science, STACS 2002*(2002).Google Scholar - [51]B. Selman, H.A. Kautz and B. Cohen, Noise strategies for improving local search, in:
*Proceedings of the 12th National Conference on Artificial Intelligence (AAAI’94)*, Seattle (1994) pp. 337–343.Google Scholar - [52]B. Selman, H. Levesque and D. Mitchell, A new method for solving hard satisfiability problems, in:
*Proceedings of the 10th National Conference on Artificial Intelligence (AAAI’92)*(1992) pp. 440–446.Google Scholar - [53]Y. Shang and B.W. Wah, A discrete Lagrangian-based global-search method for solving satisfiability problems, Journal of Global Optimization 12(1) (1998) 61–99.CrossRefzbMATHMathSciNetGoogle Scholar
- [54]L. Simon and P. Chatalic, SATEx: a Web-based framework for SAT experimentation, in: [30] (2001); http://www.lri.fr/~simon/satex.Google Scholar
- [55]G. Sutcliff and C. Suttner, Evaluating general purpose automated theorem proving systems, Artificial Intelligence 131 (2001) 39–54.CrossRefMathSciNetGoogle Scholar
- [56]G.S. Tseitin, On the complexity of derivation in the propositional calculus, in:
*Structures in Constructive Mathematics and Mathematical Logic, Part II*, ed. A.O. Slisenko (Consultants Bureau, New York, 1970) 115–125. Translated from Russian.CrossRefGoogle Scholar - [57]A. Urquhart, Hard examples for resolution, Journal of the Association for Computing Machinery 34(1) (1987) 209–219.CrossRefzbMATHMathSciNetGoogle Scholar
- [58]A. Van Gelder, Autarky pruning in propositional model elimination reduces failure redundancy, Journal of Automated Reasoning 23(2) (1999) 137–193.CrossRefzbMATHMathSciNetGoogle Scholar
- [59]A. Van Gelder, Extracting (easily) checkable proofs from a satisfiability solver that employs both pre-order and postorder resolution, in:
*Seventh Int’l Symposium on AI and Mathematics*, Fort Lauderdale, FL (2002).Google Scholar - [60]A. Van Gelder, Generalizations of watched literals for backtracking search, in:
*Seventh Int’l Symposium on AI and Mathematics*, Fort Lauderdale, FL (2002).Google Scholar - [61]A. Van Gelder and F. Okushi, Lemma and Cut strategies for propositional model elimination, Annals of Mathematics and Artificial Intelligence 26(1–4) (1999) 113–132.CrossRefzbMATHMathSciNetGoogle Scholar
- [62]A. Van Gelder and Y.K. Tsuji, Satisfiability testing with more reasoning and less guessing, in: [29, pp. 559–586] (1996).Google Scholar
- [63]M. Velev and R. Bryant, Effective use of Boolean satisfiability procedures in the formal verification of superscalar and VLIW microprocessors, in:
*Proceedings of the 38th Design Automation Conference (DAC ‘01)*(2001) pp. 226–231.Google Scholar - [64]J. Warners and H. van Maaren, Solving satisfiability problems using elliptic approximations: Effective branching rules, Discrete Applied Mathematics 107 (2000) 241–259.CrossRefzbMATHMathSciNetGoogle Scholar
- [65]H. Zhang, SATO: An efficient propositional prover, in:
*Proceedings of the International Conference on Automated Deduction (CADE’97)*, Lecture Notes in Artificial Intelligence, Vol. 1249 (1997) pp. 272-275.Google Scholar - [66]H. Zhang and M.E. Stickel, An efficient algorithm for unit propagation, in:
*Proceedings of the Fourth International Symposium on Artificial Intelligence and Mathematics (AI-MATH’96)*, Fort Lauderdale, FL (1996).Google Scholar - [67]L. Zhang, C.F. Madigan, M.W. Moskewicz and S. Malik, Efficient conflict driven learning in a Boolean satisfiability solver, in:
*International Conference on Computer-Aided Design (ICCAD’01)*(2001) pp. 279–285.Google Scholar - [68]L. Zheng and P.J. Stuckey, Improving SAT using 2SAT, in:
*Proceedings of the Twenty-Fifth Australasian Computer Science Conference (ACSC2002)*, ed. M.J. Oudshoorn, Melbourne, Australia (2002).Google Scholar

## Copyright information

© Springer 2004