Abstract
We study cdcl-cuttingplanes, Open-WBO, and Sat4j, three successful solvers from the Pseudo-Boolean Competition 2016, and evaluate them by performing experiments on crafted benchmarks designed to be trivial for the cutting planes (CP) proof system underlying pseudo-Boolean (PB) proof search but yet potentially tricky for PB solvers. Our experiments demonstrate severe shortcomings in state-of-the-art PB solving techniques. Although our benchmarks have linear-size tree-like CP proofs, and are thus extremely easy in theory, the solvers often perform quite badly even for very small instances. We believe this shows that solvers need to employ stronger rules of cutting planes reasoning. Even some instances that lack not only Boolean but also real-valued solutions are very hard in practice, which indicates that PB solvers need to get better not only at Boolean reasoning but also at linear programming. Taken together, our results point to several crucial challenges to be overcome in the quest for more efficient pseudo-Boolean solvers, and we expect that a further study of our benchmarks could shed more light on the potential and limitations of current state-of-the-art PB solving.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
An updated version of this solver with the new name RoundingSat is described in [13].
- 2.
It should be noted that [36] is closely related to the current work in that both papers are motivated by similar concerns, namely understanding the power and limitations of pseudo-Boolean reasoning. A key difference, though, is that the instances studied in [36] are designed to be potentially hard for the subsystems of cutting planes implemented by PB solvers, whereas in this paper we choose parameter settings so that almost all instances are theoretically very easy.
- 3.
A strong backdoor for an instance \(F\) to a family \(\mathcal {F}\) of (easy) instances—in this case, instances without rational solutions—is a set of variables in \(F\) such that any assignment \(\rho \) to these variables yield a restricted instance \({{F}{\upharpoonright }_{\rho }}\) that is in \(\mathcal {F}\).
- 4.
It might be worth pointing out that for an instance to lack rational solutions is the same as saying that the linear programming relaxation is infeasible, and so such instances can be shown unsatisfiable in polynomial time simply by solving the LP.
- 5.
We remark that some linearized pebbling formulas were submitted to the Pseudo-Boolean Competition 2016 under the name sumineq (sum inequalities).
- 6.
By necessity, our discussion is far from exhaustive, but readers can find all our benchmarks and the data from our experiments at http://www.csc.kth.se/~jakobn/publications/CombinatorialBenchmarksPBsolvers.
- 7.
Such a lower bound cannot be found in the literature, but is possible to obtain for graphs with good enough expansion using a variation of the techniques in [4].
- 8.
It would be interesting to verify this by a more in-depth study of Open-WBO. However, the PB version of this solver was not open-source at the time of our experiments, and also our main focus in this work is on solvers implementing CP-based reasoning.
References
Barth, P.: Linear \(0\)-\(1\) inequalities and extended clauses. Technical report MPI-I-94-216, Max-Planck-Institut für Informatik, April 1994. Preliminary version in LPAR 1993
Barth, P.: A Davis-Putnam based enumeration algorithm for linear pseudo-Boolean optimization. Technical report MPI-I-95-2-003, Max-Planck-Institut für Informatik, January 1995
Bayardo Jr., R.J., Schrag, R.: Using CSP look-back techniques to solve real-world SAT instances. In: Proceedings of the 14th National Conference on Artificial Intelligence (AAAI 1997), pp. 203–208, July 1997
Ben-Sasson, E., Wigderson, A.: Short proofs are narrow—resolution made simple. J. ACM 48(2), 149–169 (2001). Preliminary version in STOC 1999
Boros, E., Hammer, P.L.: Pseudo-Boolean optimization. Discrete Appl. Math. 123(1–3), 155–225 (2002)
Chai, D., Kuehlmann, A.: A fast pseudo-Boolean constraint solver. IEEE Trans. Comput. Aided Des. Integr. Circ. Syst. 24(3), 305–317 (2005). Preliminary version in DAC 2003
Chvátal, V.: Edmonds polytopes and a hierarchy of combinatorial problems. Discrete Math. 4(1), 305–337 (1973)
CNFgen: Combinatorial benchmarks for SAT solvers. https://github.com/MassimoLauria/cnfgen
Cook, W., Coullard, C.R., Turán, G.: On the complexity of cutting-plane proofs. Discrete Appl. Math. 18(1), 25–38 (1987)
Dixon, H.E., Ginsberg, M.L., Hofer, D.K., Luks, E.M., Parkes, A.J.: Generalizing Boolean satisfiability III: implementation. J. Artif. Intell. Res. 23, 441–531 (2005)
Eén, N., Sörensson, N.: Translating pseudo-Boolean constraints into SAT. J. Satisf. Boolean Model. Comput. 2(1–4), 1–26 (2006)
Elffers, J.: cdcl-cuttingplanes: a conflict-driven pseudo-Boolean solver (2016). Submitted to the Pseudo-Boolean Competition 2016
Elffers, J., Nordström, J.: Divide and conquer: towards faster pseudo-Boolean solving. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI-ECAI 2018), July 2018 (to appear)
Gomory, R.E.: An algorithm for integer solutions of linear programs. In: Graves, R., Wolfe, P. (eds.) Recent Advances in Mathematical Programming, pp. 269–302. McGraw-Hill, New York (1963)
Gurobi optimizer. http://www.gurobi.com/
Haken, A.: The intractability of resolution. Theor. Comput. Sci. 39(2–3), 297–308 (1985)
Heule, M., Hunt Jr., W.A., Wetzler, N.: Trimming while checking clausal proofs. In: Proceedings of the 13th International Conference on Formal Methods in Computer-Aided Design (FMCAD 2013), pp. 181–188, October 2013
Heule, M.J.H., Hunt, W.A., Wetzler, N.: Verifying refutations with extended resolution. In: Bonacina, M.P. (ed.) CADE 2013. LNCS (LNAI), vol. 7898, pp. 345–359. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38574-2_24
Joshi, S., Martins, R., Manquinho, V.: Generalized totalizer encoding for pseudo-Boolean constraints. In: Pesant, G. (ed.) CP 2015. LNCS, vol. 9255, pp. 200–209. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23219-5_15
Lauria, M., Elffers, J., Nordström, J., Vinyals, M.: CNFgen: a generator of crafted benchmarks. In: Gaspers, S., Walsh, T. (eds.) SAT 2017. LNCS, vol. 10491, pp. 464–473. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66263-3_30
Le Berre, D., Parrain, A.: The Sat4j library, release 2.2. J. Satisf. Boolean Model. Comput. 7, 59–64 (2010)
Manquinho, V.M., Marques-Silva, J.: On using cutting planes in pseudo-Boolean optimization. J. Satisf. Boolean Model. Comput. 2, 209–219 (2006). Preliminary version in SAT 2005
Manquinho, V.M., Marques-Silva, J.P.: Integration of lower bound estimates in pseudo-Boolean optimization. In: 16th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2004), pp. 742–748, November 2004
Markström, K.: Locality and hard SAT-instances. J. Satisf. Boolean Model. Comput. 2(1–4), 221–227 (2006)
Marques-Silva, J.P., Sakallah, K.A.: GRASP: a search algorithm for propositional satisfiability. IEEE Trans. Comput. 48(5), 506–521 (1999). Preliminary version in ICCAD 1996
Martins, R., Manquinho, V., Lynce, I.: Open-WBO: a modular MaxSAT solver. In: Sinz, C., Egly, U. (eds.) SAT 2014. LNCS, vol. 8561, pp. 438–445. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09284-3_33
Mikša, M., Nordström, J.: Long proofs of (seemingly) simple formulas. In: Sinz, C., Egly, U. (eds.) SAT 2014. LNCS, vol. 8561, pp. 121–137. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09284-3_10
Moskewicz, M.W., Madigan, C.F., Zhao, Y., Zhang, L., Malik, S.: Chaff: engineering an efficient SAT solver. In: Proceedings of the 38th Design Automation Conference (DAC 2001), pp. 530–535, June 2001
Open-WBO: An open source version of the MaxSAT solver WBO. http://sat.inesc-id.pt/open-wbo/
Pseudo-Boolean competition 2016, July 2016. http://www.cril.univ-artois.fr/PB16/
Roussel, O., Manquinho, V.M.: Pseudo-Boolean and cardinality constraints. In: Biere, A., Heule, M.J.H., van Maaren, H., Walsh, T. (eds.) Handbook of Satisfiability, Chap. 22. Frontiers in Artificial Intelligence and Applications, vol. 185, pp. 695–733. IOS Press, Amsterdam (2009)
Sat4j: The Boolean satisfaction and optimization library in Java. http://www.sat4j.org/
Sheini, H.M., Sakallah, K.A.: Pueblo: a hybrid pseudo-Boolean SAT solver. J. Satisf. Boolean Model. Comput. 2(1–4), 165–189 (2006). Preliminary version in DATE 2005
Spence, I.: sgen1: a generator of small but difficult satisfiability benchmarks. J. Exp. Algorithmics 15, 1.2:1–1.2:15 (2010)
Van Gelder, A., Spence, I.: Zero-one designs produce small hard SAT instances. In: Strichman, O., Szeider, S. (eds.) SAT 2010. LNCS, vol. 6175, pp. 388–397. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14186-7_37
Vinyals, M., Elffers, J., Giráldez-Cru, J., Gocht, S., Nordström, J.: In between resolution and cutting planes: a study of proof systems for pseudo-Boolean SAT solving, July 2018 (to appear)
Wetzler, N., Heule, M.J.H., Hunt, W.A.: DRAT-trim: efficient checking and trimming using expressive clausal proofs. In: Sinz, C., Egly, U. (eds.) SAT 2014. LNCS, vol. 8561, pp. 422–429. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09284-3_31
Acknowledgements
We are most grateful to Daniel Le Berre for long and patient explanations of the inner workings of pseudo-Boolean solvers, and to João Marques-Silva for helping us get an overview of relevant references for pseudo-Boolean solving. We also want to thank Ruben Martins for sharing an executable of Open-WBO with us and answering questions about the solver. Finally, we are thankful for the many detailed comments from the SAT 2018 anonymous reviewers, which helped to improve this paper considerably.
Our computational experiments were performed on resources provided by the Swedish National Infrastructure for Computing (SNIC). Many of our benchmarks were generated using the tool CNFgen [8, 20], for which we gratefully acknowledge Massimo Lauria.
The fourth author performed part of this work while at KTH Royal Institute of Technology. All authors were funded by the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007–2013)/ERC grant agreement no. 279611. The third author was also supported by Swedish Research Council grants 621-2012-5645 and 2016-00782, and the fourth author by the Prof. R Narasimhan Foundation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Elffers, J., Giráldez-Cru, J., Nordström, J., Vinyals, M. (2018). Using Combinatorial Benchmarks to Probe the Reasoning Power of Pseudo-Boolean Solvers. In: Beyersdorff, O., Wintersteiger, C. (eds) Theory and Applications of Satisfiability Testing – SAT 2018. SAT 2018. Lecture Notes in Computer Science(), vol 10929. Springer, Cham. https://doi.org/10.1007/978-3-319-94144-8_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-94144-8_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-94143-1
Online ISBN: 978-3-319-94144-8
eBook Packages: Computer ScienceComputer Science (R0)