Abstract
The complexity of constraints is a major obstacle for constraint-based software verification. Automatic constraint solvers are fundamentally incomplete: input constraints often build on some undecidable theory or some theory the solver does not support. This paper proposes and evaluates several randomized solvers to address this issue. We compared the effectiveness of a symbolic solver (CVC3), a random solver, two heuristic search solvers, and seven hybrid solvers (i.e. mix of random, symbolic, and heuristic solvers). We evaluated the solvers on a benchmark generated with a concolic execution of 9 subjects. The performance of each solver was measured by its precision, which is the fraction of constraints that the solver can find solution out of the total number of constraints that some solver can find solution. As expected, symbolic solving subsumes the other approaches for the 4 subjects that only generate decidable constraints. For the remaining 5 subjects, which contain undecidable constraints, the hybrid solvers achieved the highest precision (fraction of constraints that a solver can find a solution out of the total number of satisfiable constraints). We also observed that the solvers were complementary, which suggests that one should alternate their use in iterations of a concolic execution driver.
Similar content being viewed by others
References
Abdelbar A, Abdelshahid S (2004) Instinct-based PSO with local search applied to satisfiability. In: IEEE International Joint Conference on Neural Networks, pp 2291–2295
Anand S, Pasareanu CS, Visser W (2007) JPF-SE: a symbolic execution extension to Java PathFinder. In: Tools and algorithms for the construction and analysis of systems (TACAS), pp 134–138
Bäck T, Eiben AE, Vink ME (1998) A superior evolutionary algorithm for 3-SAT. In: 7th conference on evolutionary programming. Springer, UK, pp 125–136
Bauer A, Pister M, Tautschnig M (2007) Tool-support for the analysis of hybrid systems and models. In: Lauwereins R, Madsen J (eds) Design, automation and test in Europe conference and exposition (DATE). ACM, Nice, pp 924–929. doi:10.1145/1266366.1266565
Beizer B (1990) Software testing techniques. International Thomson Computer Press
Benhamou F, Granvilliers L (2006) Continuous and interval constraints. In: Rossi F, van Beek P, Walsh T (eds) Handbook of constraint programming, foundations of artificial intelligence, Chap. 16. Elsevier Science Publishers, Amsterdam. URL http://www.lina.sciences.univ-nantes.fr/Publications/2006/BG06
Bird DL, Munoz CU (1983) Automatic generation of random self-checking test cases. IBM Syst J 23(3): 228–245
Boyapati C, Khurshid S, Marinov D (2002) Korat: automated testing based on Java predicates. In: Proceedings of the international symposium on software testing and analysis (ISSTA), pp 123–133. doi:10.1145/566172.566191
Bozzano M, Bruttomesso R, Cimatti A, Junttila T, van Rossum P, Schulz S, Sebastiani R (2005) An incremental and layered procedure for the satisfiability of linear arithmetic logic. In: Proceedings of the international conference on tools and algorithms for the construction and analysis of systems (TACAS), pp 317–333
Csallner C, Smaragdakis Y (2004) JCrasher: an automatic robustness tester for Java. Softw Pract Exp 34: 1025–1050
Davis M, Logemann G, Loveland D (1962) A machine program for theorem-proving. Communications of ACM 5(7): 394–397. doi:10.1145/368273.368557
Dwyer MB, Elbaum S, Person S, Purandare R (2007) Parallel randomized state-space search. In: ICSE ’07: Proceedings of the 29th international conference on Software Engineering. IEEE Computer Society, Washington, DC, pp 3–12
Eiben A, van der Hauw J (1997) Solving 3-SAT by GAs adapting constraint weights. Evolutionary computation, pp 81–86. doi:10.1109/ICEC.1997.592273
Folino G, Pizzuti C, Spezzano O (1998) Combining cellular genetic algorithms and local search for solving satisfiability problems. In: IEEE conference on tools with artificial intelligence, pp 192–198
Forrester J, Miller B (2000) An empirical study of the robustness of windows NT applications using random testing. In: USENIX Windows Systems Symposium, pp 59–68
Franzle M, Herde C, Teige T, Ratschan S, Schubert T (2007) Efficient solving of large non-linear arithmetic constraint systems with complex boolean structure. J Satisf Boolean Model Comput 1: 209–236
Ganzinger H, Hagen G, Nieuwenhuis R, Oliveras A, Tinelli C (2004) DPLL(T): fast decision procedures. In: Alur R, Peled D (eds) Computer aided verification : 16th international conference, CAV 2004. Lecture Notes in Computer Science, vol 3114. Springer, Boston, pp 175–188
Godefroid P (2007) Compositional dynamic test generation. In: 34th symposium on principles of programming languages (POPL). ACM, New York, pp 47–54. doi:10.1145/1190216.1190226
Godefroid P, Klarlund N, Sen K (2005) DART: directed automated random testing. In: Proceedings of the ACM SIGPLAN conference on programming language design and implementation (PLDI), vol 40. ACM Press, New York, pp 213–223. doi:10.1145/1064978.1065036
Goldberg DE (1989) Genetic Algorithms in search, optimization and machine learning. Addison-Wesley Longman Publishing Co., Inc., Boston
Harman M (2007) Automated test data generation using search based software engineering. In: AST ’07: proceedings of the second international workshop on automation of software test, p. 2. IEEE Computer Society, Washington. doi:10.1109/AST.2007.4
Holzmann GJ, Joshi R, Groce A (2008) Swarm verification. In: Proceedings of the international conference on automated software engineering (ASE). IEEE, pp 1–6
Kennedy J, Eberhart R (1995) Particle swarm optimization. In: IEEE neural networks, pp 1942–1948
King JC (1976) Symbolic execution and program testing. Commun ACM 19(7): 385–394
Majumdar R, Sen K (2007) Hybrid concolic testing. In: Proceedings of the International Conference on Software Engineering (ICSE), pp. 416–426. doi:10.1109/ICSE.2007.41
Marchiori E, Rossi C (1999) A flipping genetic algorithm for hard 3-SAT problems. In: Genetic and evolutionary computation conference, vol 1. Morgan Kaufmann, Orlando, pp 393–400
Matsumoto M, Nishimura T (1998) Mersenne Twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans Model Comput Simul 8(1): 3–30. doi:10.1145/272991.272995
National Institute of Standards and Technology (2002) The economic impacts of inadequate infrastructure for software testing. Planning Report 02-3
Pacheco C, Ernst M (2007) Randoop: feedback-directed random testing for Java. In: Companion to the 22nd Annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications, OOPSLA, pp 815–816
Pacheco C, Ernst MD (2004) Eclat documents. Online manual. http://people.csail.mit.edu/people/cpacheco/eclat/
Pacheco C, Lahiri SK, Ernst MD, Ball T (2007) Feedback-directed random test generation. In: Proceedings of the international conference on software engineering (ICSE). IEEE Computer Society, Washington, DC, pp 75–84. doi:10.1109/ICSE.2007.37
Qadeer S (2004) Daisy file system. Joint CAV/ISSTA special event on specification, verification, and testing of concurrent software
Ranise S, Tinelli C (2006) The SMT-LIB standard: Version 1.2. Tech. rep.
Rossi C, Marchiori E, Kok JN (2000) An adaptive evolutionary algorithm for the satisfiability problem. In: Symposium on applied computing (SAC). Como, Italy, pp 463–469
Ru N, Jianhua Y (2008) A GA and particle swarm optimization based hybrid algorithm. In: IEEE world congress on computational intelligence. Hong Kong
Santhanam P, Hailpern B (2002) Software debugging, testing, and verification. IBM Syst J 41: 4–12
Sen K, Agha G (2006) CUTE and jCUTE: Concolic unit testing and explicit path model-checking tools. In: CAV, pp 419–423
Tinelli C (2002) A DPLL-based calculus for ground satisfiability modulo theories. In: Proceedings of the European conference on logics in artificial intelligence. Springer, London, pp 308–319
Windisch A, Wappler S, Wegener J (2007) Applying particle swarm optimization to software testing. In: Lipson H (ed) Genetic and evolutionary computation conference, pp 1121–1128
Wintersteiger C, Hamadi Y, de Moura L (2009) A concurrent portfolio approach to SMT solving, pp 715–720
Ypma TJ (1995) Historical development of the Newton–Raphson method. SIAM Rev 37(4): 531–551
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Takaki, M., Cavalcanti, D., Gheyi, R. et al. Randomized constraint solvers: a comparative study. Innovations Syst Softw Eng 6, 243–253 (2010). https://doi.org/10.1007/s11334-010-0124-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11334-010-0124-1