A Logic-Based Approach to Combinatorial Testing with Constraints

  • Andrea Calvagna
  • Angelo Gargantini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4966)

Abstract

Usage of combinatorial testing is wide spreading as an effective technique to reveal unintended feature interaction inside a given system. To this aim, test cases are constructed by combining tuples of assignments of the different input parameters, based on some effective combinatorial strategy. The most commonly used strategy is two-way (pairwise) coverage, requiring all combinations of valid assignments for all possible pairs of input parameters to be covered by at least one test case. In this paper a new heuristic strategy developed for the construction of pairwise covering test suites is presented, featuring a new approach to support expressive constraining over the input domain. Moreover, it allows the inclusion or exclusion of ad-hoc combinations of parameter bindings to let the user customize the test suite outcome. Our approach is tightly integrated with formal logic, since it uses test predicates to formalize combinatorial testing as a logic problem, and applies an external model checker tool to solve it. The proposed approach is supported by a prototype tool implementation, and early results of experimental assessment are also presented.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Atlee, J.M., Buckley, M.A.: A logic-model semantics for SCR software requirements. In: International Symposium on Software Testing and Analysis, ACM, New York (1996)Google Scholar
  2. 2.
    Bose, R.C., Bush, K.A.: Orthogonal arrays of strength two and three. The Annals of Mathematical Statistics 23(4), 508–524 (1952)CrossRefMathSciNetGoogle Scholar
  3. 3.
    Bryce, R.C., Colbourn, C.J.: Prioritized interaction testing for pair-wise coverage with seeding and constraints. Information & Software Technology 48(10), 960–970 (2006)CrossRefGoogle Scholar
  4. 4.
    Bryce, R.C., Colbourn, C.J., Cohen, M.B.: A framework of greedy methods for constructing interaction test suites. In: ICSE 2005, pp. 146–155. ACM, New York (2005)Google Scholar
  5. 5.
    Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: An approach to testing based on combinatorial design. IEEE Transactions On Software Engineering 23(7) (1997)Google Scholar
  6. 6.
    Cohen, M.B., Colbourn, C.J., Gibbons, P.B., Mugridge, W.B.: Constructing test suites for interaction testing. In: ICSE 2003, pp. 38–48 (2003)Google Scholar
  7. 7.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: ISSTA International symposium on Software testing and analysis, pp. 129–139. ACM Press, New York (2007)Google Scholar
  8. 8.
    Czerwonka, J.: Pairwise testing in real world. In: 24th Pacific Northwest Software Quality Conference (2006)Google Scholar
  9. 9.
    Dalal, S.R., Jain, A., Karunanithi, N., Leaton, J.M., Lott, C.M., Patton, G.C., Horowitz, B.M.: Model-based testing in practice. In: International Conference on Software Engineering ICSE, May 1999, pp. 285–295. Association for Computing Machinery, New York (1999)Google Scholar
  10. 10.
    de Moura, L., Owre, S., Rueß, H., Shankar, J.R.N., Sorea, M., Tiwari, A.: SAL 2. In: Alur, R., Peled, D.A. (eds.) CAV 2004. LNCS, vol. 3114, pp. 496–500. Springer, Heidelberg (2004)Google Scholar
  11. 11.
    Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. In: Nierstrasz, O., Lemoine, M. (eds.) ESEC 1999 and ESEC-FSE 1999. LNCS, vol. 1687, Springer, Heidelberg (1999)CrossRefGoogle Scholar
  12. 12.
    Gargantini, A., Riccobene, E.: Asm-based testing: Coverage criteria and automatic test sequence generation. JUCS 10(8) (November 2001)Google Scholar
  13. 13.
    Gargantini, A.: Using model checking to generate fault detecting tests. In: Gurevich, Y., Meyer, B. (eds.) TAP 2007. LNCS, vol. 4454, pp. 189–206. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  14. 14.
    Gargantini, A., Riccobene, E., Rinzivillo, S.: Using spin to generate tests from ASM specifications. In: Börger, E., Gargantini, A., Riccobene, E. (eds.) ASM 2003. LNCS, vol. 2589, Springer, Heidelberg (2003)CrossRefGoogle Scholar
  15. 15.
    Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies: a survey. Softw. Test, Verif. Reliab 15(3), 167–199 (2005)CrossRefGoogle Scholar
  16. 16.
    Hartman, A.: Ibm intelligent test case handler: Whitch, http://www.alphaworks.ibm.com/tech/whitch
  17. 17.
    Hartman, A.: Graph Theory, Combinatorics and Algorithms Interdisciplinary Applications. Chapter Software and Hardware Testing Using Combinatorial Covering Suites, pp. 237–266. Springer, Heidelberg (2005)Google Scholar
  18. 18.
    Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. DMATH: Discrete Mathematics 284(1-3), 149–156 (2004)MATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Hnich, B., Prestwich, S.D., Selensky, E., Smith, B.M.: Constraint models for the covering test problem. Constraints 11(2-3), 199–219 (2006)MATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Kobayashi, N., Tsuchiya, T., Kikuno, T.: Non-specification-based approaches to logic testing for software. Journal of Information and Software Technology 44(2), 113–121 (2002)CrossRefMathSciNetGoogle Scholar
  21. 21.
    Kuhn, D.R., Okum, V.: Pseudo-exhaustive testing for software. In: SEW 2006: IEEE/NASA Software Engineering Workshop, pp. 153–158. IEEE Computer Society, Los Alamitos (2006)CrossRefGoogle Scholar
  22. 22.
    Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Software Eng. 30(6), 418–421 (2004)CrossRefGoogle Scholar
  23. 23.
    Lott, C., Jain, A., Dalal, S.: Modeling requirements for combinatorial software testing. In: A-MOST 2005: Proceedings of the 1st international workshop on Advances in model-based testing, pp. 1–7. ACM Press, New York (2005)CrossRefGoogle Scholar
  24. 24.
    McDowell, A.: All-pairs testing, http://www.mcdowella.demon.co.uk/allpairs.html
  25. 25.
    Nurmela, K.: Upper bounds for covering arrays by tabu. Discrete Applied Mathematics 138(1-2), 143–152 (2004)MATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Pairwise web site, http://www.pairwise.org/
  27. 27.
    Tai, K.C., Lie, Y.: A test generation strategy for pairwise testing. IEEE Trans. Softw. Eng. 28(1), 109–111 (2002)CrossRefGoogle Scholar
  28. 28.
    TestCover tool, http://www.testcover.com/
  29. 29.
    The asmeta project, http://asmeta.sourceforge.net
  30. 30.
  31. 31.
    Williams, A.W., Probert, R.L.: A measure for component interaction test coverage. In: AICCSA, pp. 304–312. IEEE Computer Society, Los Alamitos (2001)Google Scholar
  32. 32.
    Williams, A.W.: Determination of test configurations for pair-wise interaction coverage. In: Proceedings of the 13th International Conference on the Testing of Communicating Systems (TestCom 2000), August 2000, pp. 59–74 (2000)Google Scholar
  33. 33.
    Yilmaz, C., Cohen, M.B., Porter, A.A.: Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. Software Eng. 32(1), 20–34 (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Andrea Calvagna
    • 1
  • Angelo Gargantini
    • 2
  1. 1.University of CataniaItaly
  2. 2.University of BergamoItaly

Personalised recommendations