Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  • Andrea Calvagna
  • Angelo Gargantini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5668)

Abstract

Combinatorial interaction testing aims at revealing errors inside a system under test triggered by unintended interaction between values of its input parameters. In this context we defined a new greedy approach to generate a combinatorial interaction test suites in the presence of constraints, based on integration of an SMT solver, and ordered processing of test goals. Based on the observation that the processing order of required combinations determines the size of the final test suite, this approach has been then used as a framework to evaluate a set of deterministic ordering strategies, each based on a different heuristic optimization criteria. Their performance has been assessed and contrasted also with those of random and dummy ordering strategies. Results of experimental assessment are presented and compared with well-known combinatorial tools.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Brownlie, R., Prowse, J., Phadke, M.: Robust testing of AT&T PMX/starMAIL using OATS. AT&T Technical Journal 71(3), 41–47 (1992)CrossRefGoogle Scholar
  2. 2.
    Bryce, R.C., Colbourn, C.J., Cohen, M.B.: A framework of greedy methods for constructing interaction test suites. In: ICSE 2005: Proceedings of the 27th international conference on Software engineering, pp. 146–155. ACM, New York (2005)Google Scholar
  3. 3.
    Burr, K., Young, W.: Combinatorial test techniques: Table-based automation, test generation, and code coverage. In: Proceedings of the Intl. Conf. on Software Testing Analysis and Review, October 1998, pp. 503–513 (1998)Google Scholar
  4. 4.
    Calvagna, A., Gargantini, A.: A logic-based approach to combinatorial testing with constraints. In: Beckert, B., Hähnle, R. (eds.) TAP 2008. LNCS, vol. 4966, pp. 66–83. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Calvagna, A., Gargantini, A.: Using SRI SAL model checker for combinatorial tests generation in the presence of temporal constraints. In: Rushby, J., Shankar, N. (eds.) AFM 2008: Third Workshop on Automated Formal Methods (satellite of CAV), pp. 43–52 (2008)Google Scholar
  6. 6.
    Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: An approach to testing based on combinatorial design. IEEE Transactions On Software Engineering 23(7), 437–444 (1997)CrossRefGoogle Scholar
  7. 7.
    Cohen, M.B., Colbourn, C.J., Ling, A.C.H.: Augmenting simulated annealing to build interaction test suites. In: ISSRE 2003: Proceedings of the 14th International Symposium on Software Reliability Engineering, Washington, DC, USA, p. 394. IEEE Computer Society Press, Los Alamitos (2003)CrossRefGoogle Scholar
  8. 8.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: ISSTA International symposium on Software testing and analysis, New York, NY, USA, pp. 129–139 (2007)Google Scholar
  9. 9.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Transactions on Software Engineering (to appear, 2008)Google Scholar
  10. 10.
    Czerwonka, J.: Pairwise testing in real world. In: 24th Pacific Northwest Software Quality Conference (2006)Google Scholar
  11. 11.
    Dalal, S., Jain, A., Karunanithi, N., Leaton, J., Lott, C.: Model-based testing of a highly programmable system. In: The Ninth International Symposium on Software Reliability Engineering, pp. 174–179. IEEE Computer Society, Los Alamitos (1998)Google Scholar
  12. 12.
    Dalal, S.R., Jain, A., Karunanithi, N., Leaton, J.M., Lott, C.M., Patton, G.C., Horowitz, B.M.: Model-based testing in practice. In: International Conference on Software Engineering ICSE, pp. 285–295. Association for Computing Machinery, New York (1999)Google Scholar
  13. 13.
    Dunietz, I.S., Ehrlich, W.K., Szablak, B., Mallows, C., Iannino, A.: Applying design of experiments to software testing. In: Society, I. (ed.) Proc. Int’l. Conf. Software Eng. (ICSE), pp. 205–215 (1997)Google Scholar
  14. 14.
    Dutertre, B., de Moura, L.: The Yices SMT solver. Technical report, SRI (2006), http://yices.csl.sri.com/tool-paper.pdf
  15. 15.
    Fraser, G., Gargantini, A., Wotawa, F.: On the order of test goals in specification-based testing. Journal of Logic and Algebraic Programming (in press, 2009)Google Scholar
  16. 16.
    Gargantini, A.: Using model checking to generate fault detecting tests. In: Gurevich, Y., Meyer, B. (eds.) TAP 2007. LNCS, vol. 4454, pp. 189–206. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  17. 17.
    Gargantini, A., Riccobene, E.: Asm-based testing: Coverage criteria and automatic test sequence generation. JUCS 10(8) (November 2001)Google Scholar
  18. 18.
    Gargantini, A., Riccobene, E., Rinzivillo, S.: Using spin to generate tests from ASM specifications. In: Börger, E., Gargantini, A., Riccobene, E. (eds.) ASM 2003. LNCS, vol. 2589, pp. 263–277. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  19. 19.
    Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies: a survey. Softw. Test, Verif. Reliab. 15(3), 167–199 (2005)CrossRefGoogle Scholar
  20. 20.
    Hartman, A.: IBM intelligent test case handler: WhitchGoogle Scholar
  21. 21.
    Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. DMATH: Discrete Mathematics 284(1-3), 149–156 (2004)MathSciNetMATHGoogle Scholar
  22. 22.
    Hnich, B., Prestwich, S.D., Selensky, E., Smith, B.M.: Constraint models for the covering test problem. Constraints 11(2-3), 199–219 (2006)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Kuhn, D.R., Okum, V.: Pseudo-exhaustive testing for software. In: SEW 2006: IEEE/NASA Software Engineering Workshop, vol. 0, pp. 153–158. IEEE Computer Society Press, Los Alamitos (2006)CrossRefGoogle Scholar
  24. 24.
    Kuhn, D.R., Reilly, M.J.: An investigation of the applicability of design of experiments to software testing. In: 27th NASA/IEEE Software Engineering workshop, pp. 91–95. IEEE Computer Society, Los Alamitos (2002)Google Scholar
  25. 25.
    Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Software Eng. 30(6), 418–421 (2004)CrossRefGoogle Scholar
  26. 26.
    Lott, C., Jain, A., Dalal, S.: Modeling requirements for combinatorial software testing. In: A-MOST 2005: Proceedings of the 1st international workshop on Advances in model-based testing, pp. 1–7. ACM Press, New York (2005)Google Scholar
  27. 27.
    Pairwise web site, http://www.pairwise.org/
  28. 28.
    Sherwood, G.B.: Optimal and near-optimal mixed covering arrays by column expansion. Discrete Mathematics 308(24), 6022–6035 (2008)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Smith, B.D., Feather, M.S., Muscettola, N.: Challenges and methods in validating the remote agent planner. In: Breckenridge, C. (ed.) Fifth International conference on Artificial Intelligence Planning Systems (AIPS) (2000)Google Scholar
  30. 30.
    TestCover tool, http://www.testcover.com/
  31. 31.
    Walsh, T.: SAT v CSP. In: Dechter, R. (ed.) CP 2000. LNCS, vol. 1894, pp. 441–456. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  32. 32.
    Wong, W.E., Horgan, J.R., Mathur, A.P., Pasquini, A.: Test set size minimization and fault detection effectiveness: A case study in a space application. Journal of Systems and Software 48(2), 79–89 (1999)CrossRefGoogle Scholar
  33. 33.
    Yilmaz, C., Cohen, M.B., Porter, A.A.: Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. Software Eng. 32(1), 20–34 (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Andrea Calvagna
    • 1
  • Angelo Gargantini
    • 2
  1. 1.Dip. Ingegneria Informatica e delle TelecomunicazioniUniversity of CataniaItaly
  2. 2.Dip. Metodi Matematici e Ingegneria dell’InformazioneUniversity of BergamoItaly

Personalised recommendations