Innovations in Systems and Software Engineering

, Volume 6, Issue 3, pp 243–253 | Cite as

Randomized constraint solvers: a comparative study

  • Mitsuo Takaki
  • Diego Cavalcanti
  • Rohit Gheyi
  • Juliano Iyoda
  • Marcelo d’Amorim
  • Ricardo B. C. Prudêncio
Original Paper

Abstract

The complexity of constraints is a major obstacle for constraint-based software verification. Automatic constraint solvers are fundamentally incomplete: input constraints often build on some undecidable theory or some theory the solver does not support. This paper proposes and evaluates several randomized solvers to address this issue. We compared the effectiveness of a symbolic solver (CVC3), a random solver, two heuristic search solvers, and seven hybrid solvers (i.e. mix of random, symbolic, and heuristic solvers). We evaluated the solvers on a benchmark generated with a concolic execution of 9 subjects. The performance of each solver was measured by its precision, which is the fraction of constraints that the solver can find solution out of the total number of constraints that some solver can find solution. As expected, symbolic solving subsumes the other approaches for the 4 subjects that only generate decidable constraints. For the remaining 5 subjects, which contain undecidable constraints, the hybrid solvers achieved the highest precision (fraction of constraints that a solver can find a solution out of the total number of satisfiable constraints). We also observed that the solvers were complementary, which suggests that one should alternate their use in iterations of a concolic execution driver.

Keywords

Constraint solvers Software testing Concolic execution Random solvers 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Abdelbar A, Abdelshahid S (2004) Instinct-based PSO with local search applied to satisfiability. In: IEEE International Joint Conference on Neural Networks, pp 2291–2295Google Scholar
  2. 2.
    Anand S, Pasareanu CS, Visser W (2007) JPF-SE: a symbolic execution extension to Java PathFinder. In: Tools and algorithms for the construction and analysis of systems (TACAS), pp 134–138Google Scholar
  3. 3.
    Bäck T, Eiben AE, Vink ME (1998) A superior evolutionary algorithm for 3-SAT. In: 7th conference on evolutionary programming. Springer, UK, pp 125–136Google Scholar
  4. 4.
    Bauer A, Pister M, Tautschnig M (2007) Tool-support for the analysis of hybrid systems and models. In: Lauwereins R, Madsen J (eds) Design, automation and test in Europe conference and exposition (DATE). ACM, Nice, pp 924–929. doi: 10.1145/1266366.1266565
  5. 5.
    Beizer B (1990) Software testing techniques. International Thomson Computer PressGoogle Scholar
  6. 6.
    Benhamou F, Granvilliers L (2006) Continuous and interval constraints. In: Rossi F, van Beek P, Walsh T (eds) Handbook of constraint programming, foundations of artificial intelligence, Chap. 16. Elsevier Science Publishers, Amsterdam. URL http://www.lina.sciences.univ-nantes.fr/Publications/2006/BG06
  7. 7.
    Bird DL, Munoz CU (1983) Automatic generation of random self-checking test cases. IBM Syst J 23(3): 228–245Google Scholar
  8. 8.
    Boyapati C, Khurshid S, Marinov D (2002) Korat: automated testing based on Java predicates. In: Proceedings of the international symposium on software testing and analysis (ISSTA), pp 123–133. doi: 10.1145/566172.566191
  9. 9.
    Bozzano M, Bruttomesso R, Cimatti A, Junttila T, van Rossum P, Schulz S, Sebastiani R (2005) An incremental and layered procedure for the satisfiability of linear arithmetic logic. In: Proceedings of the international conference on tools and algorithms for the construction and analysis of systems (TACAS), pp 317–333Google Scholar
  10. 10.
    Csallner C, Smaragdakis Y (2004) JCrasher: an automatic robustness tester for Java. Softw Pract Exp 34: 1025–1050CrossRefGoogle Scholar
  11. 11.
    Davis M, Logemann G, Loveland D (1962) A machine program for theorem-proving. Communications of ACM 5(7): 394–397. doi: 10.1145/368273.368557 MATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Dwyer MB, Elbaum S, Person S, Purandare R (2007) Parallel randomized state-space search. In: ICSE ’07: Proceedings of the 29th international conference on Software Engineering. IEEE Computer Society, Washington, DC, pp 3–12Google Scholar
  13. 13.
    Eiben A, van der Hauw J (1997) Solving 3-SAT by GAs adapting constraint weights. Evolutionary computation, pp 81–86. doi: 10.1109/ICEC.1997.592273
  14. 14.
    Folino G, Pizzuti C, Spezzano O (1998) Combining cellular genetic algorithms and local search for solving satisfiability problems. In: IEEE conference on tools with artificial intelligence, pp 192–198Google Scholar
  15. 15.
    Forrester J, Miller B (2000) An empirical study of the robustness of windows NT applications using random testing. In: USENIX Windows Systems Symposium, pp 59–68Google Scholar
  16. 16.
    Franzle M, Herde C, Teige T, Ratschan S, Schubert T (2007) Efficient solving of large non-linear arithmetic constraint systems with complex boolean structure. J Satisf Boolean Model Comput 1: 209–236Google Scholar
  17. 17.
    Ganzinger H, Hagen G, Nieuwenhuis R, Oliveras A, Tinelli C (2004) DPLL(T): fast decision procedures. In: Alur R, Peled D (eds) Computer aided verification : 16th international conference, CAV 2004. Lecture Notes in Computer Science, vol 3114. Springer, Boston, pp 175–188Google Scholar
  18. 18.
    Godefroid P (2007) Compositional dynamic test generation. In: 34th symposium on principles of programming languages (POPL). ACM, New York, pp 47–54. doi: 10.1145/1190216.1190226
  19. 19.
    Godefroid P, Klarlund N, Sen K (2005) DART: directed automated random testing. In: Proceedings of the ACM SIGPLAN conference on programming language design and implementation (PLDI), vol 40. ACM Press, New York, pp 213–223. doi: 10.1145/1064978.1065036
  20. 20.
    Goldberg DE (1989) Genetic Algorithms in search, optimization and machine learning. Addison-Wesley Longman Publishing Co., Inc., BostonGoogle Scholar
  21. 21.
    Harman M (2007) Automated test data generation using search based software engineering. In: AST ’07: proceedings of the second international workshop on automation of software test, p. 2. IEEE Computer Society, Washington. doi: 10.1109/AST.2007.4
  22. 22.
    Holzmann GJ, Joshi R, Groce A (2008) Swarm verification. In: Proceedings of the international conference on automated software engineering (ASE). IEEE, pp 1–6Google Scholar
  23. 23.
    Kennedy J, Eberhart R (1995) Particle swarm optimization. In: IEEE neural networks, pp 1942–1948Google Scholar
  24. 24.
    King JC (1976) Symbolic execution and program testing. Commun ACM 19(7): 385–394MATHCrossRefGoogle Scholar
  25. 25.
    Majumdar R, Sen K (2007) Hybrid concolic testing. In: Proceedings of the International Conference on Software Engineering (ICSE), pp. 416–426. doi: 10.1109/ICSE.2007.41
  26. 26.
    Marchiori E, Rossi C (1999) A flipping genetic algorithm for hard 3-SAT problems. In: Genetic and evolutionary computation conference, vol 1. Morgan Kaufmann, Orlando, pp 393–400Google Scholar
  27. 27.
    Matsumoto M, Nishimura T (1998) Mersenne Twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans Model Comput Simul 8(1): 3–30. doi: 10.1145/272991.272995 MATHCrossRefGoogle Scholar
  28. 28.
    National Institute of Standards and Technology (2002) The economic impacts of inadequate infrastructure for software testing. Planning Report 02-3Google Scholar
  29. 29.
    Pacheco C, Ernst M (2007) Randoop: feedback-directed random testing for Java. In: Companion to the 22nd Annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications, OOPSLA, pp 815–816Google Scholar
  30. 30.
    Pacheco C, Ernst MD (2004) Eclat documents. Online manual. http://people.csail.mit.edu/people/cpacheco/eclat/
  31. 31.
    Pacheco C, Lahiri SK, Ernst MD, Ball T (2007) Feedback-directed random test generation. In: Proceedings of the international conference on software engineering (ICSE). IEEE Computer Society, Washington, DC, pp 75–84. doi: 10.1109/ICSE.2007.37
  32. 32.
    Qadeer S (2004) Daisy file system. Joint CAV/ISSTA special event on specification, verification, and testing of concurrent softwareGoogle Scholar
  33. 33.
    Ranise S, Tinelli C (2006) The SMT-LIB standard: Version 1.2. Tech. rep.Google Scholar
  34. 34.
    Rossi C, Marchiori E, Kok JN (2000) An adaptive evolutionary algorithm for the satisfiability problem. In: Symposium on applied computing (SAC). Como, Italy, pp 463–469Google Scholar
  35. 35.
    Ru N, Jianhua Y (2008) A GA and particle swarm optimization based hybrid algorithm. In: IEEE world congress on computational intelligence. Hong KongGoogle Scholar
  36. 36.
    Santhanam P, Hailpern B (2002) Software debugging, testing, and verification. IBM Syst J 41: 4–12CrossRefGoogle Scholar
  37. 37.
    Sen K, Agha G (2006) CUTE and jCUTE: Concolic unit testing and explicit path model-checking tools. In: CAV, pp 419–423Google Scholar
  38. 38.
    Tinelli C (2002) A DPLL-based calculus for ground satisfiability modulo theories. In: Proceedings of the European conference on logics in artificial intelligence. Springer, London, pp 308–319Google Scholar
  39. 39.
    Windisch A, Wappler S, Wegener J (2007) Applying particle swarm optimization to software testing. In: Lipson H (ed) Genetic and evolutionary computation conference, pp 1121–1128Google Scholar
  40. 40.
    Wintersteiger C, Hamadi Y, de Moura L (2009) A concurrent portfolio approach to SMT solving, pp 715–720Google Scholar
  41. 41.
    Ypma TJ (1995) Historical development of the Newton–Raphson method. SIAM Rev 37(4): 531–551MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2010

Authors and Affiliations

  • Mitsuo Takaki
    • 1
  • Diego Cavalcanti
    • 2
  • Rohit Gheyi
    • 2
  • Juliano Iyoda
    • 1
  • Marcelo d’Amorim
    • 1
  • Ricardo B. C. Prudêncio
    • 1
  1. 1.Federal University of PernambucoRecifeBrazil
  2. 2.Federal University of Campina GrandeCampina GrandeBrazil

Personalised recommendations