Skip to main content
Log in

Abstract

The annual Satisfiability Modulo Theories Competition (SMT-COMP) was initiated in 2005 in order to stimulate the advance of state-of-the-art techniques and tools developed by the Satisfiability Modulo Theories (SMT) community. This paper summarizes the first six editions of the competition. We present the evolution of the competition’s organization and rules, show how the state of the art has improved over the course of the competition, and discuss the impact SMT-COMP has had on the SMT community and beyond. Additionally, we include an exhaustive list of all competitors, and present experimental results showing significant improvement in SMT solvers during these six years. Finally, we analyze to what extent the initial goals of the competition have been achieved, and sketch future directions for the competition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Barrett, C.: Checking validity of quantifier-free formulas in combinations of first-order theories. PhD thesis, Stanford University (2003)

  2. Barrett, C., Berezin, S.: CVC Lite: a new implementation of the cooperating validity checker. In: Alur, R., Peled, D.A. (eds.) Proceedings of the 16th International Conference on Computer Aided Verification (CAV ’04). Lecture Notes in Computer Science, vol. 3114, pp. 515–518. Springer, Boston, Massachusetts (2004)

    Chapter  Google Scholar 

  3. Brummayer, R., Biere, A.: Fuzzing and delta-debugging SMT solvers. In: Strichman, O., Dutertre, B. (eds.) 7th International Workshop on Satisfiability Modulo Theories (2009)

  4. Brummayer, R., Biere, A.: Boolector: an efficient smt solver for bit-vectors and arrays. In: Kowalewski, S., Philippou, A. (eds.) 15th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’09. Lecture Notes in Computer Science, vol. 5505, pp. 174–177. Springer (2009)

  5. Bozzano, M., Bruttomesso, R., Cimatti, A., Junttila, T.A., Ranise, S., van Rossum, P., Sebastiani, R.: Efficient satisfiability modulo theories via delayed theory combination. In: Proceedings of the 17th International Conference on Computer Aided Verification (CAV’05). Lecture Notes in Computer Science. Springer (2005)

  6. Brummayer, R., Biere, A., Lonsing, F.: BTOR: bit-precise modelling of word-level problems for model checking. In: SMT ’08/BPR ’08: Proceedings of the Joint Workshops of the 6th International Workshop on Satisfiability Modulo Theories and 1st International Workshop on Bit-Precise Reasoning, pp. 33–38. ACM, New York, NY, USA (2008)

    Chapter  Google Scholar 

  7. Barrett, C., Dill, D.L., Levitt, J.R.: Validity checking for combinations of theories with equality. In: Srivas, M., Camilleri, A. (eds.) Proceedings of the 1st International Conference on Formal Methods In Computer-Aided Design (FMCAD ’96). Lecture Notes in Computer Science, vol. 1166, pp. 187–201. Springer, Palo Alto, California (1996)

    Chapter  Google Scholar 

  8. Biere, A.: PicoSAT essentials. JSAT 4(2–4), 75–97 (2008)

    MATH  Google Scholar 

  9. Biere, A.: P{re,i}coSAT@SC’09. http://fmv.jku.at/precosat/preicosat-sc09.pdf (2009)

  10. Borralleras, C., Lucas, S., Navarro-Marset, R., Rodríguez-Carbonell, E., Rubio, A.: Solving non-linear polynomial arithmetic via SAT modulo linear arithmetic. In: Schmidt, R.A. (ed.) 22nd International Conference on Automated Deduction , CADE-22. Lecture Notes in Computer Science, vol. 5663, pp. 294–305. Springer (2009)

  11. Bofill, M., Nieuwenhuis, R., Oliveras, A., Rodríguez-Carbonell, E., Rubio, A.: The barcelogic SMT solver. In: Gupta, A., Malik, S. (eds.) Computer Aided Verification (CAV), pp. 294–298. Springer (2008)

  12. Barrett, C.W., Sebastiani, R., Seshia, S.A., Tinelli, C.: Satisfiability modulo theories. In: Biere, A., Heule, M., van Maaren, H., Walsh, T. (eds.) Handbook of Satisfiability. Frontiers in Artificial Intelligence and Applications, vol. 185, pp. 825–885. IOS Press (2009)

  13. Barrett, C., Stump, A., Tinelli, C.: The SMT-LIB standard—version 2.0. In: Proceedings of the 8th International Workshop on Satisfiability Modulo Theories (SMT ’10). Edinburgh, Scotland (2010)

  14. Conchon, S., Contejean, E., Kanig, J., Lescuyer, S.: CC(X): semantic combination of congruence closure with solvable theories. Electron. Notes Theor. Comput. Sci. 198(2), 51–69 (2008)

    Article  MathSciNet  Google Scholar 

  15. Cotton, S., Maler, O.: Fast and flexible difference constraint propagation for DPLL(T). In: Biere, A., Gomes, C.P. (eds.) 9th International Conference on Theory and Applications of Satisfiability Testing, SAT’06. Lecture Notes in Computer Science, vol. 4121, pp. 170–183. Springer (2006)

  16. Dutertre, B., de Moura, L.: A fast linear-arithmetic solver for DPLL(T). In: Ball, T., Jones, R.B. (eds.) Proceedings of the 18th International Conference on Computer Aided Verification (CAV’06). Lecture Notes in Computer Science, vol. 4144, pp. 81–94. Springer (2006)

  17. Dutertre, B., De Moura, L.: Integrating simplex with DPLL(T). Technical report, CSL, SRI International (2006)

  18. de Moura, L., Bjørner, N.: Efficient E-matching for SMT solvers. In: Pfenning, F. (ed.) 21st International Conference on Automated Deduction, CADE-21. Lecture Notes in Computer Science, vol. 4603, pp. 183–198. Springer (2007)

  19. de Moura, L., Bjørner, N.: Model-based theory combination. Electron. Notes Theor. Comput. Sci. 198(2), 37–49 (2008)

    Article  Google Scholar 

  20. de Moura, L., Ruess, H.: An experimental evaluation of ground decision procedures. In: Alur, R., Peled, D. (eds.) 16th International Conference on Computer Aided Verification, CAV’04. Lecture Notes in Computer Science, vol. 3114, pp. 162–174. Springer (2004)

  21. Eén, N., Sörensson, N.: An extensible SAT-solver. In: Proceedings of the Sixth International Conference on Theory and Applications of Satisfiability Testing (SAT 2003). Lecture Notes in Computer Science, vol. 2919, pp. 502–518. Springer (2003)

  22. Fuhs, C., Giesl, J., Middeldorp, A., Schneider-Kamp, P., Thiemann, R., Zankl, H.: SAT solving for termination analysis with polynomial interpretations. In: Marques-Silva, J., Sakallah, K.A. (eds.) 10th International Conference on Theory and Applications of Satisfiability Testing, SAT’07. Lecture Notes in Computer Science, vol. 4501, pp. 340–354. Springer (2007)

  23. Filliâtre, J., Owre, S., Rueß, H., Shankar, N.: ICS: integrated canonizer and solver. In: Berry, G., Comon, H., Finkel, A. (eds.) 13th International Conference on Computer-Aided Verification (2001)

  24. Ge, Y., Barrett, C., Tinelli, C.: Solving quantified verification conditions using satisfiability modulo theories. Ann. Math. Artif. Intell. 55(1–2), 101–122 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  25. Ganzinger, H., Hagen, G., Nieuwenhuis, R., Oliveras, A., Tinelli, C.: DPLL(T): fast decision procedures. In: Proceedings of the 16th International Conference on Computer Aided Verification (CAV’04). Lecture Notes in Computer Science, vol. 3114, pp. 175–188. Springer (2004)

  26. Griggio, A.: A practical approach to SMT(LA(Z)). In: 8th International Workshop on Satisfiability Modulo Theories (2010)

  27. Ganai, M.K., Talupur, M., Gupta, A.: SDSAT: tight Integration of small domain encoding and lazy approaches in solving difference logic. JSAT 3(1–2), 91–114 (2007)

    MATH  Google Scholar 

  28. Jain, H., Clarke, E.M.: Efficient SAT solving for non-clausal formulas using DPLL, graphs, and watched cuts. In: 46th Design Automation Conference (DAC) (2009)

  29. Jin, H., Han, H., Somenzi, F.: Efficient conflict analysis for finding all satisfying assignments of a boolean circuit. In: In TACAS’05. LNCS, vol. 3440, pp. 287–300. Springer (2005)

  30. Kim, H., Somenzi, F., Jin, H.: Efficient term-ITE conversion for satisfiability modulo theories. In: Kullmann, O. (ed.) 12th International Conference on Theory and Applications of Satisfiability Testing, SAT’09. Lecture Notes in Computer Science, vol. 4121, pp. 195–208. Springer (2009)

  31. Le Berre, D., Simon, L.: Preface, special issue on the SAT 2005 competitions and evaluations. JSAT 2, 1–4 (2006)

    Google Scholar 

  32. Levitt, J.: Formal verification techniques for digital systems. PhD thesis, Stanford University (1999)

  33. Moskal, M., Lopuszanski, J., Kiniry, J.R.: E-matching for fun and profit. Electron. Notes Theor. Comput. Sci. 198(2), 19–35 (2008)

    Article  MathSciNet  Google Scholar 

  34. Moskewicz, M., Madigan, C., Zhaod, Y., Zhang, L., Malik, S.: Chaff: engineering an efficient SAT solver. In: 39th Design Automation Conference (2001)

  35. Nikolić, M.: Statistical methodology for comparison of SAT solvers. In: Strichman, O., Szeider, S. (eds.) Thirteenth International Conference on Theory and Applications of Satisfiability Testing (SAT) (2010)

  36. Nelson, G., Oppen, D.: Simplification by cooperating decision procedures. TOPLAS 1(2), 245–57 (1979)

    Article  MATH  Google Scholar 

  37. Nieuwenhuis, R., Oliveras, A.: DPLL(T) with exhaustive theory propagation and its application to difference logic. In: Proceedings of the 17th International Conference on Computer Aided Verification (CAV’05). Lecture Notes in Computer Science. Springer (2005)

  38. Pipatsrisawat, K., Darwiche, A.: Rsat 2.0: SAT solver description. Technical Report D–153, Automated Reasoning Group, Computer Science Department, UCLA. http://reasoning.cs.ucla.edu/rsat/papers/rsat_2.0.pdf (2007)

  39. Rueß, H., Shankar, N.: Solving linear arithmetic constraints. Technical Report SRI-CSL-04-01, SRI International (2004)

  40. Stump, A., Barrett, C.W., Dill, D.L.: CVC: a cooperating validity checker. In: 14th International Conference on Computer-Aided Verification, pp. 500–504. Springer (2002)

  41. Simon, L., Le Berre, D., Hirsch, E.A.: The SAT2002 competition. Ann. Math. Artif. Intell. 43(1), 307–342 (2005)

    Article  Google Scholar 

  42. Shankar, N.: Little engines of proof. In: Eriksson, L.H., Lindsay, P.A. (eds.) International Symposium of Formal Methods Europe, FME’02. Lecture Notes in Computer Science, vol. 2391, pp. 1–20. Springer (2002)

  43. Sutcliffe, G., Suttner, C.: The state of CASC. AI Commun. 19(1), 35–48 (2006)

    MathSciNet  MATH  Google Scholar 

  44. Sutcliffe, G.: The TPTP problem library and associated infrastructure. J. Autom. Reasoning 43(4), 337–362 (2009)

    Article  MATH  Google Scholar 

  45. Wang, B.: On the satisfiability of modular arithmetic formulae. In: Graf, S., Zhang, W. (eds.) 4th International Symposium of Automated Techonology for Verification and Analysis, ATVA’06. Lecture Notes in Computer Science, vol. 4218, pp. 186–199. Springer (2006)

  46. Wang, C., Ivancic, F., Ganai, M.K., Gupta, A.: Deciding separation logic formulae by SAT and incremental negative cycle elimination. In: Sutcliffe, G., Voronkov, A. (eds.) 12h International Conference on Logic for Programming, Artificial Intelligence and Reasoning, LPAR’05. Lecture Notes in Computer Science, vol. 3835, pp. 322–336. Springer (2005)

  47. Williams, H.P.: Fourier-Motzkin elimination extension to integer programming problems. J. Combin. Theory Ser. A 21, 118–123 (1976)

    Article  MATH  Google Scholar 

  48. Zhang, H.: SATO: an efficient propositional prover. In: McCune, W. (ed.) Proceedings of the 14th International Conference on Automated deduction. Lecture Notes in Artificial Intelligence, vol. 1249, pp. 272–275. Springer (1997)

  49. Zankl, H., Middeldorp, A.: Satisfiability of non-linear (ir)rational arithmetic. In: Clarke, E.M., Voronkov, A. (eds.) 16th International Conference on Logic for Programming, Artificial Intelligence and Reasoning, LPAR’10. Lecture Notes in Computer Science, vol. 6355, pp. 481–500. Springer (2010)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Albert Oliveras.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Barrett, C., Deters, M., de Moura, L. et al. 6 Years of SMT-COMP. J Autom Reasoning 50, 243–277 (2013). https://doi.org/10.1007/s10817-012-9246-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10817-012-9246-5

Keywords

Navigation