CNFgen: A Generator of Crafted Benchmarks

  • Massimo Lauria
  • Jan Elffers
  • Jakob Nordström
  • Marc Vinyals
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10491)


We present CNFgen, a generator of combinatorial benchmarks in DIMACS and OPB format. The proof complexity literature is a rich source not only of hard instances but also of instances that are theoretically easy but “extremal” in different ways, and therefore of potential interest in the context of SAT solving. Since most of these formulas appear not to be very well known in the SAT community, however, we propose CNFgen as a resource to make them readily available for solver development and evaluation. Many formulas studied in proof complexity are based on graphs, and CNFgen is also able to generate, parse and do basic manipulation of such objects. Furthermore, it includes a library cnfformula giving access to the functionality of CNFgen to Python programs.



The first author performed most of this work while at KTH Royal Institute of Technology. The authors were funded by the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007–2013) / ERC grant agreement no. 279611 as well as by Swedish Research Council grant 621-2012-5645. The first author was also supported by the European Research Council under the European Union’s Horizon 2020 Research and Innovation Programme / ERC grant agreement no. 648276 AUTAR.


  1. 1.
    Alekhnovich, M., Ben-Sasson, E., Alexander, A., Razborov, A.A., Wigderson, A.: Space complexity in propositional calculus. SIAM J. Comput. 31(4), 1184–1211 (2002). Preliminary version in STOC ’00MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Alekhnovich, M., Johannsen, J., Pitassi, T., Urquhart, A.: An exponential separation between regular and general resolution. Theor. Comput. 3(5), 81–102 (2007). Preliminary version in STOC ’02MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Alekhnovich, M., Alexander, A. Razborov, A.A.: Lower bounds for polynomial calculus: Non-binomial case. In: Proceedings of the Steklov Institute of Mathematics, 242, 18–35 (2003). Preliminary version in FOCS ’01
  4. 4.
    Balyo, T., Marijn, J., Heule, H., Järvisalo, M.: Proceedings of SAT competition 2016: Solver and benchmark descriptions. Technical report B-2016-1, University of Helsinki (2016).
  5. 5.
    Roberto, J., Bayardo, Jr., Schrag, R.: Using CSP look-back techniques to solve real-world SAT instances. In Proceedings of the 14th National Conference on Artificial Intelligence (AAAI 1997), pp. 203–208 (1997)Google Scholar
  6. 6.
    Beame, P., Beck, C., Impagliazzo, R.: Time-space tradeoffs in resolution: Superpolynomial lower bounds for superlinear space. In: Proceedings of the 44th Annual ACM Symposium on Theory of Computing (STOC 2012), pp. 213–232 (2012)Google Scholar
  7. 7.
    Beck, C., Nordström, J., Tang, B.: Some trade-off results for polynomial calculus. In: Proceedings of the 45th Annual ACM Symposium on Theory of Computing (STOC 2013), pp. 813–822 (2013)Google Scholar
  8. 8.
    Ben-Sasson, E., Nordström, J.: Short proofs may be spacious: An optimal separation of space and length in resolution. In: Proceedings of the 49th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2008), pp. 709–718 (2008)Google Scholar
  9. 9.
    Ben-Sasson, E., Nordström, J.: Understanding space in proof complexity: Separations and trade-offs via substitutions. In: Proceedings of the 2nd Symposium on Innovations in Computer Science (ICS 2011), pp. 401–416 (2011)Google Scholar
  10. 10.
    Beyersdorff, O., Galesi, N., Lauria, M.: Parameterized complexity of DPLL search procedures. ACM Trans. Comput. Logic 14(3), 20:1–20:21 (2013). Preliminary version in SAT ’11MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Beyersdorff, O., Galesi, N., Lauria, M., Razborov, A.A.: Parameterized bounded-depth Frege is not optimal. ACM Trans. Comput. Theor. 4, 7:1–7:16 (2012). Preliminary version in ICALP ’11CrossRefzbMATHGoogle Scholar
  12. 12.
    Blake, A.: Canonical expressions in boolean algebra. PhD thesis, University of Chicago (1937)Google Scholar
  13. 13.
    Bonet, M.L., Galesi, N.: Optimality of size-width tradeoffs for resolution. Comput. Complex. 10(4), 261–276 (2001). Preliminary version in FOCS ’99MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Brickenstein, M., Dreyer, A.: PolyBoRi: A framework for Gröbner-basis computations with Boolean polynomials. J. Symb. Comput. 44(9), 1326–1345 (2009)CrossRefzbMATHGoogle Scholar
  15. 15.
    Brickenstein, M., Dreyer, A., Greuel, G.-M., Wedler, M., Wienand, O.: New developments in the theory of Gröbner bases and applications to formal verification. J. Pure Appl. Algebr. 213(8), 1612–1635 (2009)CrossRefzbMATHGoogle Scholar
  16. 16.
    Buss, S.R., Grigoriev, D., Impagliazzo, R., Pitassi, T.: Linear gaps between degrees for the polynomial calculus modulo distinct primes. J. Comput. Syst. Sci. 62(2), 267–289 (2001). Preliminary version in CCC ’99MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Buss, S.R., Kołodziejczyk, L.: Small stone in pool. Logic. Method. Comput. Sci. 10, 10:16–16:22 (2014)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Chai, D., Kuehlmann, A.: A fast pseudo-Boolean constraint solver. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 24(3), 305–317 (2005). Preliminary version in DAC ’03CrossRefGoogle Scholar
  19. 19.
    Chvátal, V., Szemerédi, E.: Many hard examples for resolution. J. ACM 35(4), 759–768 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Clegg, M., Edmonds, J., Impagliazzo, R.: Using the Groebner basis algorithm to find proofs of unsatisfiability. In: Proceedings of the 28th Annual ACM Symposium on Theory of Computing (STOC 1996), pp. 174–183 (1996)Google Scholar
  21. 21.
    Cook, S.A.: The complexity of theorem-proving procedures. In: Proceedings of the 3rd Annual ACM Symposium on Theory of Computing (STOC 1971), pp. 151–158 (1971)Google Scholar
  22. 22.
    Cook, S.A., Reckhow, R.: The relative efficiency of propositional proof systems. J. Symbol. Logic 44(1), 36–50 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Cook, W.: Collette Rene Coullard, and György Turán. On the complexity of cutting-plane proofs. Discr. Appl. Math. 18(1), 25–38 (1987)CrossRefGoogle Scholar
  24. 24.
    Dixon, H.E., Ginsberg, M.L., Hofer, D.K., Luks, E.M., Parkes, A.J.: Generalizing Boolean satisfiability III: Implementation. J. Artif. Intell. Res. 23, 441–531 (2005)zbMATHGoogle Scholar
  25. 25.
    Elffers, J., Giráldez-Crú, J., Nordström, J., Vinyals, M.: Using combinatorial benchmarks to probe the reasoning power of pseudo-Boolean solvers (2017, Submitted)Google Scholar
  26. 26.
    Elffers, J., Nordström, J., Simon, L., Sakallah, K.A.: Seeking practical CDCL insights from theoretical SAT benchmarks. In: Presentation at the Pragmatics of SAT 2016 workshop (2016).
  27. 27.
    Galesi, N., Lauria, M.: Optimality of size-degree trade-offs for polynomial calculus. ACM Trans. Comput. Logic 12, 4:1–4:22 (2010)CrossRefzbMATHGoogle Scholar
  28. 28.
    Hagberg, A.A., Schult, D., Swart, P.S.: Exploring network structure, dynamics, and function using NetworkX. In: Proceedings of the 7th Python in Science Conference (SciPy2008), Pasadena, CA USA, pp. 11–15 (2008)Google Scholar
  29. 29.
    Haken, A.: The intractability of resolution. Theor. Comput. Sci. 39(2–3), 297–308 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Heule, M., van Maaren, H.: Aligning CNF- and equivalence-reasoning. In: Hoos, H.H., Mitchell, D.G. (eds.) SAT 2004. LNCS, vol. 3542, pp. 145–156. Springer, Heidelberg (2005). doi: 10.1007/11527695_12 CrossRefGoogle Scholar
  31. 31.
    Himsolt, M.: GML: A portable graph file format. Technical report, Universität of Passau (1996)Google Scholar
  32. 32.
    Huynh, T., Nordström, J.: On the virtue of succinct proofs: Amplifying communication complexity hardness to time-space trade-offs in proof complexity (Extended abstract). In: Proceedings of the 44th Annual ACM Symposium on Theory of Computing (STOC 2012), pp. 233–248 (2012)Google Scholar
  33. 33.
    Impagliazzo, R., Paturi, R.: On the complexity of \(k\)-SAT. J. Comput. Syst. Sci. 62(2), 367–375 (2001). Preliminary version in CCC ’99MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Järvisalo, M., Matsliah, A., Nordström, J., Živný, S.: Relating proof complexity measures and practical hardness of SAT. In: Milano, M. (ed.) CP 2012. LNCS, pp. 316–331. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33558-7_25 CrossRefGoogle Scholar
  35. 35.
    Le Berre, D., Parrain, A.: The Sat4j library, release 2.2. J. Satisf. Boolean Model. Comput. 7, 59–64 (2010)Google Scholar
  36. 36.
    Markström, K.: Locality and hard SAT-instances. J. Satisf. Boolean Model. Comput. 2(1–4), 221–227 (2006)zbMATHGoogle Scholar
  37. 37.
    Marques-Silva, J.P., Sakallah, K.A.: GRASP: A search algorithm for propositional satisfiability. IEEE Trans. Comput. 48(5), 506–521 (1999). Preliminary version in ICCAD ’96MathSciNetCrossRefGoogle Scholar
  38. 38.
    Massey, B.: DIMACS graph format (2001). Accessed 11 Feb 2016
  39. 39.
    Mikša, M., Nordström, J.: Long proofs of (seemingly) simple formulas. In: Sinz, C., Egly, U. (eds.) SAT 2014. LNCS, vol. 8561, pp. 121–137. Springer, Cham (2014). doi: 10.1007/978-3-319-09284-3_10 Google Scholar
  40. 40.
    Mikša, M., Nordström, J.: A generalized method for proving polynomial calculus degree lower bounds. In: Proceedings of the 30th Annual Computational Complexity Conference (CCC 2015). Leibniz International Proceedings in Informatics (LIPIcs), vol. 33, pp. 467–487 (2015)Google Scholar
  41. 41.
    Moskewicz, M.W., Madigan, C.F., Zhao, Y., Zhang, L., Chaff, M.S.: Engineering an efficient SAT solver. In: Proceedings of the 38th Design Automation Conference (DAC 2001), pp. 530–535 (2001)Google Scholar
  42. 42.
    Nordström, J.: On the interplay between proof complexity and SAT solving. ACM SIGLOG News 2(3), 19–44 (2015)Google Scholar
  43. 43.
    Pseudo-Boolean competition (2016).
  44. 44.
    Razborov, A.A.: Resolution lower bounds for perfect matching principles. J. Comput. Syst. Sci. 69(1), 3–27 (2004). Preliminary version in CCC ’02MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Razborov, A.A.: A new kind of tradeoffs in propositional proof complexity. J. ACM 63, 16:1–16:14 (2016)MathSciNetCrossRefGoogle Scholar
  46. 46.
    AT and T Research: Dot Language. Accessed 11 Feb 2016
  47. 47.
    Sheini, H.M., Sakallah, K.A.: Pueblo: a hybrid pseudo-Boolean SAT solver. J. Satisf. Boolean Model. Comput. 2(1–4), 165–189 (2006). Preliminary version in DATE ’05zbMATHGoogle Scholar
  48. 48.
    Soos, M., Nohl, K., Castelluccia, C.: Extending SAT solvers to cryptographic problems. In: Kullmann, O. (ed.) SAT 2009. LNCS, vol. 5584, pp. 244–257. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-02777-2_24 CrossRefGoogle Scholar
  49. 49.
    Stålmarck, G.: Short resolution proofs for a sequence of tricky formulas. Acta Inform. 33(3), 277–280 (1996)MathSciNetCrossRefGoogle Scholar
  50. 50.
    Urquhart, A.: Hard examples for resolution. J. ACM 34(1), 209–219 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  51. 51.
    Gelder, A., Spence, I.: Zero-one designs produce small hard SAT instances. In: Strichman, O., Szeider, S. (eds.) SAT 2010. LNCS, vol. 6175, pp. 388–397. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-14186-7_37 CrossRefGoogle Scholar
  52. 52.
    Vinyals, M., Elffers, J., Giráldez-Crú, J., Gocht, S., Nordström, J.: In between resolution and cutting planes: A study of proof systems for pseudo-Boolean SAT solving (2017, Submitted)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Massimo Lauria
    • 1
  • Jan Elffers
    • 2
  • Jakob Nordström
    • 2
  • Marc Vinyals
    • 2
  1. 1.Università degli studi di Roma “La Sapienza”RomeItaly
  2. 2.KTH Royal Institute of TechnologyStockholmSweden

Personalised recommendations