Journal of Automated Reasoning

, Volume 50, Issue 3, pp 243–277

6 Years of SMT-COMP

  • Clark Barrett
  • Morgan Deters
  • Leonardo de Moura
  • Albert Oliveras
  • Aaron Stump
Article

Abstract

The annual Satisfiability Modulo Theories Competition (SMT-COMP) was initiated in 2005 in order to stimulate the advance of state-of-the-art techniques and tools developed by the Satisfiability Modulo Theories (SMT) community. This paper summarizes the first six editions of the competition. We present the evolution of the competition’s organization and rules, show how the state of the art has improved over the course of the competition, and discuss the impact SMT-COMP has had on the SMT community and beyond. Additionally, we include an exhaustive list of all competitors, and present experimental results showing significant improvement in SMT solvers during these six years. Finally, we analyze to what extent the initial goals of the competition have been achieved, and sketch future directions for the competition.

Keywords

SAT Modulo Theories Competition Experimental evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Barrett, C.: Checking validity of quantifier-free formulas in combinations of first-order theories. PhD thesis, Stanford University (2003)Google Scholar
  2. 2.
    Barrett, C., Berezin, S.: CVC Lite: a new implementation of the cooperating validity checker. In: Alur, R., Peled, D.A. (eds.) Proceedings of the 16th International Conference on Computer Aided Verification (CAV ’04). Lecture Notes in Computer Science, vol. 3114, pp. 515–518. Springer, Boston, Massachusetts (2004)CrossRefGoogle Scholar
  3. 3.
    Brummayer, R., Biere, A.: Fuzzing and delta-debugging SMT solvers. In: Strichman, O., Dutertre, B. (eds.) 7th International Workshop on Satisfiability Modulo Theories (2009)Google Scholar
  4. 4.
    Brummayer, R., Biere, A.: Boolector: an efficient smt solver for bit-vectors and arrays. In: Kowalewski, S., Philippou, A. (eds.) 15th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’09. Lecture Notes in Computer Science, vol. 5505, pp. 174–177. Springer (2009)Google Scholar
  5. 5.
    Bozzano, M., Bruttomesso, R., Cimatti, A., Junttila, T.A., Ranise, S., van Rossum, P., Sebastiani, R.: Efficient satisfiability modulo theories via delayed theory combination. In: Proceedings of the 17th International Conference on Computer Aided Verification (CAV’05). Lecture Notes in Computer Science. Springer (2005)Google Scholar
  6. 6.
    Brummayer, R., Biere, A., Lonsing, F.: BTOR: bit-precise modelling of word-level problems for model checking. In: SMT ’08/BPR ’08: Proceedings of the Joint Workshops of the 6th International Workshop on Satisfiability Modulo Theories and 1st International Workshop on Bit-Precise Reasoning, pp. 33–38. ACM, New York, NY, USA (2008)CrossRefGoogle Scholar
  7. 7.
    Barrett, C., Dill, D.L., Levitt, J.R.: Validity checking for combinations of theories with equality. In: Srivas, M., Camilleri, A. (eds.) Proceedings of the 1st International Conference on Formal Methods In Computer-Aided Design (FMCAD ’96). Lecture Notes in Computer Science, vol. 1166, pp. 187–201. Springer, Palo Alto, California (1996)CrossRefGoogle Scholar
  8. 8.
    Biere, A.: PicoSAT essentials. JSAT 4(2–4), 75–97 (2008)MATHGoogle Scholar
  9. 9.
    Biere, A.: P{re,i}coSAT@SC’09. http://fmv.jku.at/precosat/preicosat-sc09.pdf (2009)
  10. 10.
    Borralleras, C., Lucas, S., Navarro-Marset, R., Rodríguez-Carbonell, E., Rubio, A.: Solving non-linear polynomial arithmetic via SAT modulo linear arithmetic. In: Schmidt, R.A. (ed.) 22nd International Conference on Automated Deduction , CADE-22. Lecture Notes in Computer Science, vol. 5663, pp. 294–305. Springer (2009)Google Scholar
  11. 11.
    Bofill, M., Nieuwenhuis, R., Oliveras, A., Rodríguez-Carbonell, E., Rubio, A.: The barcelogic SMT solver. In: Gupta, A., Malik, S. (eds.) Computer Aided Verification (CAV), pp. 294–298. Springer (2008)Google Scholar
  12. 12.
    Barrett, C.W., Sebastiani, R., Seshia, S.A., Tinelli, C.: Satisfiability modulo theories. In: Biere, A., Heule, M., van Maaren, H., Walsh, T. (eds.) Handbook of Satisfiability. Frontiers in Artificial Intelligence and Applications, vol. 185, pp. 825–885. IOS Press (2009)Google Scholar
  13. 13.
    Barrett, C., Stump, A., Tinelli, C.: The SMT-LIB standard—version 2.0. In: Proceedings of the 8th International Workshop on Satisfiability Modulo Theories (SMT ’10). Edinburgh, Scotland (2010)Google Scholar
  14. 14.
    Conchon, S., Contejean, E., Kanig, J., Lescuyer, S.: CC(X): semantic combination of congruence closure with solvable theories. Electron. Notes Theor. Comput. Sci. 198(2), 51–69 (2008)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Cotton, S., Maler, O.: Fast and flexible difference constraint propagation for DPLL(T). In: Biere, A., Gomes, C.P. (eds.) 9th International Conference on Theory and Applications of Satisfiability Testing, SAT’06. Lecture Notes in Computer Science, vol. 4121, pp. 170–183. Springer (2006)Google Scholar
  16. 16.
    Dutertre, B., de Moura, L.: A fast linear-arithmetic solver for DPLL(T). In: Ball, T., Jones, R.B. (eds.) Proceedings of the 18th International Conference on Computer Aided Verification (CAV’06). Lecture Notes in Computer Science, vol. 4144, pp. 81–94. Springer (2006)Google Scholar
  17. 17.
    Dutertre, B., De Moura, L.: Integrating simplex with DPLL(T). Technical report, CSL, SRI International (2006)Google Scholar
  18. 18.
    de Moura, L., Bjørner, N.: Efficient E-matching for SMT solvers. In: Pfenning, F. (ed.) 21st International Conference on Automated Deduction, CADE-21. Lecture Notes in Computer Science, vol. 4603, pp. 183–198. Springer (2007)Google Scholar
  19. 19.
    de Moura, L., Bjørner, N.: Model-based theory combination. Electron. Notes Theor. Comput. Sci. 198(2), 37–49 (2008)CrossRefGoogle Scholar
  20. 20.
    de Moura, L., Ruess, H.: An experimental evaluation of ground decision procedures. In: Alur, R., Peled, D. (eds.) 16th International Conference on Computer Aided Verification, CAV’04. Lecture Notes in Computer Science, vol. 3114, pp. 162–174. Springer (2004)Google Scholar
  21. 21.
    Eén, N., Sörensson, N.: An extensible SAT-solver. In: Proceedings of the Sixth International Conference on Theory and Applications of Satisfiability Testing (SAT 2003). Lecture Notes in Computer Science, vol. 2919, pp. 502–518. Springer (2003)Google Scholar
  22. 22.
    Fuhs, C., Giesl, J., Middeldorp, A., Schneider-Kamp, P., Thiemann, R., Zankl, H.: SAT solving for termination analysis with polynomial interpretations. In: Marques-Silva, J., Sakallah, K.A. (eds.) 10th International Conference on Theory and Applications of Satisfiability Testing, SAT’07. Lecture Notes in Computer Science, vol. 4501, pp. 340–354. Springer (2007)Google Scholar
  23. 23.
    Filliâtre, J., Owre, S., Rueß, H., Shankar, N.: ICS: integrated canonizer and solver. In: Berry, G., Comon, H., Finkel, A. (eds.) 13th International Conference on Computer-Aided Verification (2001)Google Scholar
  24. 24.
    Ge, Y., Barrett, C., Tinelli, C.: Solving quantified verification conditions using satisfiability modulo theories. Ann. Math. Artif. Intell. 55(1–2), 101–122 (2009)MathSciNetMATHCrossRefGoogle Scholar
  25. 25.
    Ganzinger, H., Hagen, G., Nieuwenhuis, R., Oliveras, A., Tinelli, C.: DPLL(T): fast decision procedures. In: Proceedings of the 16th International Conference on Computer Aided Verification (CAV’04). Lecture Notes in Computer Science, vol. 3114, pp. 175–188. Springer (2004)Google Scholar
  26. 26.
    Griggio, A.: A practical approach to SMT(LA(Z)). In: 8th International Workshop on Satisfiability Modulo Theories (2010)Google Scholar
  27. 27.
    Ganai, M.K., Talupur, M., Gupta, A.: SDSAT: tight Integration of small domain encoding and lazy approaches in solving difference logic. JSAT 3(1–2), 91–114 (2007)MATHGoogle Scholar
  28. 28.
    Jain, H., Clarke, E.M.: Efficient SAT solving for non-clausal formulas using DPLL, graphs, and watched cuts. In: 46th Design Automation Conference (DAC) (2009)Google Scholar
  29. 29.
    Jin, H., Han, H., Somenzi, F.: Efficient conflict analysis for finding all satisfying assignments of a boolean circuit. In: In TACAS’05. LNCS, vol. 3440, pp. 287–300. Springer (2005)Google Scholar
  30. 30.
    Kim, H., Somenzi, F., Jin, H.: Efficient term-ITE conversion for satisfiability modulo theories. In: Kullmann, O. (ed.) 12th International Conference on Theory and Applications of Satisfiability Testing, SAT’09. Lecture Notes in Computer Science, vol. 4121, pp. 195–208. Springer (2009)Google Scholar
  31. 31.
    Le Berre, D., Simon, L.: Preface, special issue on the SAT 2005 competitions and evaluations. JSAT 2, 1–4 (2006)Google Scholar
  32. 32.
    Levitt, J.: Formal verification techniques for digital systems. PhD thesis, Stanford University (1999)Google Scholar
  33. 33.
    Moskal, M., Lopuszanski, J., Kiniry, J.R.: E-matching for fun and profit. Electron. Notes Theor. Comput. Sci. 198(2), 19–35 (2008)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Moskewicz, M., Madigan, C., Zhaod, Y., Zhang, L., Malik, S.: Chaff: engineering an efficient SAT solver. In: 39th Design Automation Conference (2001)Google Scholar
  35. 35.
    Nikolić, M.: Statistical methodology for comparison of SAT solvers. In: Strichman, O., Szeider, S. (eds.) Thirteenth International Conference on Theory and Applications of Satisfiability Testing (SAT) (2010)Google Scholar
  36. 36.
    Nelson, G., Oppen, D.: Simplification by cooperating decision procedures. TOPLAS 1(2), 245–57 (1979)MATHCrossRefGoogle Scholar
  37. 37.
    Nieuwenhuis, R., Oliveras, A.: DPLL(T) with exhaustive theory propagation and its application to difference logic. In: Proceedings of the 17th International Conference on Computer Aided Verification (CAV’05). Lecture Notes in Computer Science. Springer (2005)Google Scholar
  38. 38.
    Pipatsrisawat, K., Darwiche, A.: Rsat 2.0: SAT solver description. Technical Report D–153, Automated Reasoning Group, Computer Science Department, UCLA. http://reasoning.cs.ucla.edu/rsat/papers/rsat_2.0.pdf (2007)
  39. 39.
    Rueß, H., Shankar, N.: Solving linear arithmetic constraints. Technical Report SRI-CSL-04-01, SRI International (2004)Google Scholar
  40. 40.
    Stump, A., Barrett, C.W., Dill, D.L.: CVC: a cooperating validity checker. In: 14th International Conference on Computer-Aided Verification, pp. 500–504. Springer (2002)Google Scholar
  41. 41.
    Simon, L., Le Berre, D., Hirsch, E.A.: The SAT2002 competition. Ann. Math. Artif. Intell. 43(1), 307–342 (2005)CrossRefGoogle Scholar
  42. 42.
    Shankar, N.: Little engines of proof. In: Eriksson, L.H., Lindsay, P.A. (eds.) International Symposium of Formal Methods Europe, FME’02. Lecture Notes in Computer Science, vol. 2391, pp. 1–20. Springer (2002)Google Scholar
  43. 43.
    Sutcliffe, G., Suttner, C.: The state of CASC. AI Commun. 19(1), 35–48 (2006)MathSciNetMATHGoogle Scholar
  44. 44.
    Sutcliffe, G.: The TPTP problem library and associated infrastructure. J. Autom. Reasoning 43(4), 337–362 (2009)MATHCrossRefGoogle Scholar
  45. 45.
    Wang, B.: On the satisfiability of modular arithmetic formulae. In: Graf, S., Zhang, W. (eds.) 4th International Symposium of Automated Techonology for Verification and Analysis, ATVA’06. Lecture Notes in Computer Science, vol. 4218, pp. 186–199. Springer (2006)Google Scholar
  46. 46.
    Wang, C., Ivancic, F., Ganai, M.K., Gupta, A.: Deciding separation logic formulae by SAT and incremental negative cycle elimination. In: Sutcliffe, G., Voronkov, A. (eds.) 12h International Conference on Logic for Programming, Artificial Intelligence and Reasoning, LPAR’05. Lecture Notes in Computer Science, vol. 3835, pp. 322–336. Springer (2005)Google Scholar
  47. 47.
    Williams, H.P.: Fourier-Motzkin elimination extension to integer programming problems. J. Combin. Theory Ser. A 21, 118–123 (1976)MATHCrossRefGoogle Scholar
  48. 48.
    Zhang, H.: SATO: an efficient propositional prover. In: McCune, W. (ed.) Proceedings of the 14th International Conference on Automated deduction. Lecture Notes in Artificial Intelligence, vol. 1249, pp. 272–275. Springer (1997)Google Scholar
  49. 49.
    Zankl, H., Middeldorp, A.: Satisfiability of non-linear (ir)rational arithmetic. In: Clarke, E.M., Voronkov, A. (eds.) 16th International Conference on Logic for Programming, Artificial Intelligence and Reasoning, LPAR’10. Lecture Notes in Computer Science, vol. 6355, pp. 481–500. Springer (2010)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  • Clark Barrett
    • 1
  • Morgan Deters
    • 1
  • Leonardo de Moura
    • 2
  • Albert Oliveras
    • 3
  • Aaron Stump
    • 4
  1. 1.New York UniversityNew YorkUSA
  2. 2.Microsoft ResearchRedmondUSA
  3. 3.Technical University of CataloniaBarcelonaSpain
  4. 4.The University of IowaIowa CityUSA

Personalised recommendations