Journal of Automated Reasoning

, Volume 45, Issue 4, pp 331–358 | Cite as

A Formal Logic Approach to Constrained Combinatorial Testing

  • Andrea Calvagna
  • Angelo GargantiniEmail author


Combinatorial testing is as an effective testing technique to reveal failures in a given system, based on input combinations coverage and combinatorial optimization. Combinatorial testing of strength t (t ≥ 2) requires that each t-wise tuple of values of the different system input parameters is covered by at least one test case. Combinatorial test suite generation algorithms aim at producing a test suite covering all the required tuples in a small (possibly minimal) number of test cases, in order to reduce the cost of testing. The most used combinatorial technique is the pairwise testing (t = 2) which requires coverage of all pairs of input values. Constrained combinatorial testing takes also into account constraints over the system parameters, for instance forbidden tuples of inputs, modeling invalid or not realizable input values combinations. In this paper a new approach to combinatorial testing, tightly integrated with formal logic, is presented. In this approach, test predicates are used to formalize combinatorial testing as a logical problem, and an external formal logic tool is applied to solve it. Constraints over the input domain are expressed as logical predicates too, and effectively handled by the same tool. Moreover, inclusion or exclusion of select tuples is supported, allowing the user to customize the test suite layout. The proposed approach is supported by a prototype tool implementation and results of experimental assessment are also presented.


Model-based testing Combinatorial testing Test generation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Atlee, J.M., Buckley, M.A.: A logic-model semantics for SCR software requirements. In: International Symposium on Software Testing and Analysis. ACM (1996)Google Scholar
  2. 2.
    Biere, A., Cimatti, A., Clarke, E.M., Zhu, Y.: Symbolic model checking without bdds. In: In TACAS 99: Proceedings of the 5th International Conference on Tools and Algorithms for Construction and Analysis of Systems, London, UK, 1999, pp. 193–207. Springer, Heidelberg, ISBN 3-540-65703-7 (1999)Google Scholar
  3. 3.
    Blass, A., Gurevich, Y.: Pairwise testing. In: Current Trends in Theoretical Computer Science, pp. 237–266. World Scientific Press, Singapore (2004)CrossRefGoogle Scholar
  4. 4.
    Bose, R.C., Bush, K.A.: Orthogonal arrays of strength two and three. Ann. Math. Stat. 23(4), 508–524 (1952)zbMATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Brownlie, R., Prowse, J., Phadke, M.: Robust testing of AT&T PMX/starMAIL using OATS. ATT Tech. J. 71(3), 41–47 (1992)Google Scholar
  6. 6.
    Bryce, R.C., Colbourn, C.J.: Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inf. Softw. Technol. 48(10), 960–970 (2006)CrossRefGoogle Scholar
  7. 7.
    Burr, K., Young, W.: Combinatorial test techniques: table-based automation, test generation, and code coverage. In: Proceedings of the Intl. Conf. on Software Testing Analysis and Review, pp. 503–513 (1998)Google Scholar
  8. 8.
    Calvagna, A., Gargantini, A.: A logic-based approach to combinatorial testing with constraints. In: Beckert, B., Hähnle, R. (eds.) Tests and Proofs, Second International Conference, TAP 2008, Prato, Italy, 9–11 April 2008. Proceedings. Lecture Notes in Computer Science, vol. 4966, pp. 66–83. Springer, Heidelberg (2008)Google Scholar
  9. 9.
    Calvagna, A., Gargantini, A.: Using SRI SAL model checker for combinatorial tests generation in the presence of temporal constraints. In: Rushby, J., Shankar, N. (eds.) AFM’08: Third Workshop on Automated Formal Methods (satellite of CAV), pp. 43–52 (2008)Google Scholar
  10. 10.
    Chvatal, V.: A greedy heuristic for the set-covering problem. Math. Oper. Res. 4(3), 233–235 (1979)zbMATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23(7), 437–444 (1997)CrossRefGoogle Scholar
  12. 12.
    Cohen, M., Dwyer, M., Shi, J.: Exploiting constraint solving history to construct interaction test suites. In: Testing: Academic and Industrial Conference-Practice and Research Techniques (TAIC PART), London, September 2007, pp. 121–130 (2007)Google Scholar
  13. 13.
    Cohen, M.B., Colbourn, C.J., Gibbons, P.B., Mugridge, W.B.: Constructing test suites for interaction testing. In: ICSE, pp. 38–48 (2003)Google Scholar
  14. 14.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: ISSTA International Symposium on Software Testing and Analysis, pp. 129–139. ACM Press, New York, NY, USA (2007)CrossRefGoogle Scholar
  15. 15.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34, 633–650 (2010)CrossRefGoogle Scholar
  16. 16.
    Czerwonka, J.: Pairwise testing in real world. In: 24th Pacific Northwest Software Quality Conference (2006)Google Scholar
  17. 17.
    Dalal, S., Jain, A., Karunanithi, N., Leaton, J., Lott, C.: Model-based testing of a highly programmable system. ISSRE Proceedings 1998, 5–7 November, pp. 174–178 (1998)Google Scholar
  18. 18.
    Dalal, S.R., Jain, A., Karunanithi, N., Leaton, J.M., Lott, C.M., Patton, G.C., Horowitz, B.M.: Model-based testing in practice. In: International Conference on Software Engineering ICSE, pp. 285–295. Association for Computing Machinery, New York (1999)CrossRefGoogle Scholar
  19. 19.
    de Moura, L., Owre, S., Rueß, H., Rushby, J., Shankar, N., Sorea, M., Tiwari, A.: SAL 2. In: Alur, R., Peled, D. (eds.) Computer-Aided Verification, CAV 2004. Lecture Notes in Computer Science, vol. 3114, pp. 496–500. Springer, Boston, MA (2004)CrossRefGoogle Scholar
  20. 20.
    Dunietz, I.S., Ehrlich, W.K., Szablak, B., Mallows, C., Iannino, A.: Applying design of experiments to software testing. In: Society, I. (ed.) Proc. Int’l Conf. Software Eng. (ICSE), pp. 205–215 (1997)Google Scholar
  21. 21.
    Fraser, G., Wotawa, F.: Using LTL rewriting to improve the performance of model-checker based test-case generation. In: Proceedings of the 3rd International Workshop on Advances in Model Based Testing (A-MOST 2007), pp. 64–74 (2007)Google Scholar
  22. 22.
    Fraser, G., Wotawa, F.: Ordering coverage goals in model checker based testing. In: Proceedings of the Fourth International Workshop on Advances in Model Based Testing (A-MOST 2008) (2008)Google Scholar
  23. 23.
    Gargantini, A.: Using model checking to generate fault detecting tests. In: International Conference on Tests And Proofs (TAP). Lecture Notes in Computer Science (LNCS), vol. 4454, pp. 189–206. Springer, Heidelberg (2007)Google Scholar
  24. 24.
    Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. In: Nierstrasz, O., Lemoine, M. (eds.) Software Engineering ESEC/FSE’99. LNCS, vol. 1687 (1999)Google Scholar
  25. 25.
    Gargantini, A., Riccobene, E.: Asm-based testing: coverage criteria and automatic test sequence generation. JUCS 7(11), 1050–1067 (2001)Google Scholar
  26. 26.
    Gargantini, A., Riccobene, E., Rinzivillo, S.: Using spin to generate tests from ASM specifications. In: ASM 2003 - Taormina, Italy, March 2003. Proceedings. LNCS, vol. 2589 (2003)Google Scholar
  27. 27.
    Gargantini, A., Riccobene, E., Scandurra, P.: A metamodel-based language and a simulation engine for abstract state machines. JUCS 14, 1949–1983 (2008)Google Scholar
  28. 28.
    Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies: a survey. Softw. Test. Verif. Reliab. 15(3), 167–199 (2005)CrossRefGoogle Scholar
  29. 29.
    Hamon, G., de Moura, L., Rushby, J.: Generating efficient test sets with a model checker. In: 2nd International Conference on Software Engineering and Formal Methods, pp. 261–270. IEEE Computer Society, Beijing, China (2004)Google Scholar
  30. 30.
    Hartman, A.: Ibm intelligent test case handler: whitch.
  31. 31.
    Hartman, A.: Graph theory, combinatorics and algorithms interdisciplinary applications. In: chap. Software and Hardware Testing Using Combinatorial Covering Suites, pp. 237–266. Springer (2005)Google Scholar
  32. 32.
    Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. Discrete Math. 284(1–3), 149–156 (2004)zbMATHCrossRefMathSciNetGoogle Scholar
  33. 33.
    Hnich, B., Prestwich, S.D., Selensky, E., Smith, B.M.: Constraint models for the covering test problem. Constraints 11(2–3), 199–219 (2006)zbMATHCrossRefMathSciNetGoogle Scholar
  34. 34.
    Holzmann, G.J.: The model checker SPIN. Softw. Eng. 23(5), 279–295 (1997)CrossRefMathSciNetGoogle Scholar
  35. 35.
  36. 36.
    Kobayashi, N., Tsuchiya, T., Kikuno, T.: Non-specification-based approaches to logic testing for software. J. Inf. Softw. Technol. 44(2), 113–121 (2002)CrossRefMathSciNetGoogle Scholar
  37. 37.
    Kuhn, D.R., Okum, V.: Pseudo-exhaustive testing for software. In: SEW ’06: IEEE/NASA Software Engineering Workshop, vol. 0, pp. 153–158. IEEE Computer Society, Los Alamitos, CA, USA (2006)CrossRefGoogle Scholar
  38. 38.
    Kuhn, D.R., Reilly, M.J.: An investigation of the applicability of design of experiments to software testing. In: Society, I. (ed.) 27th NASA/IEEE Software Engineering Workshop, pp. 91–95 (2002)Google Scholar
  39. 39.
    Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30(6), 418–421 (2004)CrossRefGoogle Scholar
  40. 40.
    Lott, C., Jain, A., Dalal, S.: Modeling requirements for combinatorial software testing. In: A-MOST ’05: Proceedings of the 1st International Workshop on Advances in Model-Based Testing, pp. 1–7. ACM Press, New York, NY, USA (2005)CrossRefGoogle Scholar
  41. 41.
    McDowell, A.: All-pairs testing.
  42. 42.
    McMillan, K.L.: Symbolic Model Checking. Kluwer Academic Publishers, Norwell, MA, USA, ISBN 0792393805 (1993)zbMATHGoogle Scholar
  43. 43.
    Nurmela, K.: Upper bounds for covering arrays by tabu. Discrete Appl. Math. 138(1–2), 143–152 (2004)zbMATHCrossRefMathSciNetGoogle Scholar
  44. 44.
    Pairwise web site:
  45. 45.
    Seroussi, G., Bshouty, N.H.: Vector sets for exhaustive testing of logic circuits. IEEE Trans. Inf. Theory 34(3), 513–522 (1988)zbMATHCrossRefMathSciNetGoogle Scholar
  46. 46.
    Smith, B.D., Feather, M.S., Muscettola, N.: Challenges and methods in validating the remote agent planner. In: Breckenridge, C. (ed.) Proceedings of the Fifth International Conference on Artificial Intelligence Planning Systems (AIPS) (2000)Google Scholar
  47. 47.
    Tai, K.C., Lie, Y.: A test generation strategy for pairwise testing. IEEE Trans. Softw. Eng. 28(1), 109–111 (2002)CrossRefGoogle Scholar
  48. 48.
    Williams, A.: Determination of test configurations for pair-wise interaction coverage. In: Proceedings of the 13th International Conference on the Testing of Communicating Systems (TestCom 2000), pp. 59–74 (2000)Google Scholar
  49. 49.
    Williams, A.W., Probert, R.L.: Formulation of the interaction test coverage problem as an integer program. In: Proceedings of the 14th International Conference on the Testing of Communicating Systems (TestCom) Berlin, Germany, pp. 283–298 (2002)Google Scholar
  50. 50.
    Yilmaz, C., Cohen, M.B., Porter, A.A.: Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. Softw. Eng. 32(1), 20–34 (2006)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  1. 1.Dip. Ingegneria Informatica e delle TelecomunicazioniUniversity of CataniaCataniaItaly
  2. 2.Dip. Metodi Matematici e Ingegneria dell’InformazioneUniversity of BergamoBergamoItaly

Personalised recommendations