Efficient Combinatorial Test Generation Based on Multivalued Decision Diagrams

  • Angelo Gargantini
  • Paolo Vavassori
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8855)

Abstract

Combinatorial interaction testing (CIT) is an emerging testing technique that has proved to be effective in finding faults due to the interaction among inputs. Efficient test generation for CIT is still an open problem especially when applied to real models having meaningful size and containing many constraints among inputs. In this paper we present a novel technique for the automatic generation of compact test suites starting from models containing constraints given in general form. It is based on the use of Multivalued Decision Diagrams (MDDs) which prove to be suitable to efficiently support CIT. We devise and experiment several optimizations including a novel variation of the classical greedy policy normally used in similar algorithms. The results of a thorough comparison with other similar techniques are presented and show that our approach can provide several advantages in terms of applicability, test suite size, generation time, and cost.

Keywords

Test Suite Test Generation Terminal Node Software Product Line Binary Decision Diagram 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Advanced Combinatorial Testing System (ACTS), http://csrc.nist.gov/groups/SNS/acts/
  2. 2.
    Arcaini, P., Gargantini, A., Vavassori, P.: Validation of models and tests for constrained combinatorial interaction testing. In: The 3rd International Workshop on Combinatorial Testing (IWCT 2014) In conjunction with International Conference on Software Testing ICSTW, pp. 98–107. IEEE (2014)Google Scholar
  3. 3.
    Babar, J., Miner, A.: Meddly: Multi-terminal and edge-valued decision diagram library. In: 7th International Conference on the Quantitative Evaluation of Systems. IEEE (2010)Google Scholar
  4. 4.
    Brownlie, R., Prowse, J., Phadke, M.: Robust testing of AT&T PMX/starMAIL using OATS. AT&T Technical Journal 71(3), 41–47 (1992)CrossRefGoogle Scholar
  5. 5.
    Bryce, R.C., Colbourn, C.J.: Prioritized interaction testing for pair-wise coverage with seeding and constraints. Information & Software Technology 48(10), 960–970 (2006)CrossRefGoogle Scholar
  6. 6.
    Bryce, R.C., Colbourn, C.J.: One-test-at-a-time heuristic search for interaction test suites. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, GECCO 2007, pp. 1082–1089. ACM, New York (2007)Google Scholar
  7. 7.
    Bryce, R.C., Colbourn, C.J., Cohen, M.B.: A framework of greedy methods for constructing interaction test suites. In: ICSE 2005: Proc. of the 27th Int. Conf. on Software Engineering, pp. 146–155. ACM, New York (2005)Google Scholar
  8. 8.
    Calvagna, A., Gargantini, A.: A formal logic approach to constrained combinatorial testing. Journal of Automated Reasoning 45(4), 331–358 (2010)CrossRefMATHMathSciNetGoogle Scholar
  9. 9.
    Calvagna, A., Gargantini, A.: T-wise combinatorial interaction test suites construction based on coverage inheritance. Software Testing, Verification and Reliability 22(7), 507–526 (2012)CrossRefGoogle Scholar
  10. 10.
    Calvagna, A., Gargantini, A., Vavassori, P.: Combinatorial interaction testing with CitLab. In: Sixth IEEE International Conference on Software Testing, Verification and Validation - Testing Tool Track (2013)Google Scholar
  11. 11.
    Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: An approach to testing based on combinatorial design. IEEE Transactions On Software Engineering 23(7), 437–444 (1997)CrossRefGoogle Scholar
  12. 12.
    Cohen, M., Dwyer, M., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Trans. on Software Engineering 34(5), 633–650 (2008)CrossRefGoogle Scholar
  13. 13.
    Covering Arrays by Simulated Annealing, http://cse.unl.edu/citportal/tools/casa/
  14. 14.
    Garvin, B.J., Cohen, M.B., Dwyer, M.B.: An improved meta-heuristic search for constrained interaction testing. In: Proceedings of the 2009 1st International Symposium on Search Based Software Engineering, SSBSE 2009, pp. 13–22. IEEE Computer Society, Washington, DC (2009)CrossRefGoogle Scholar
  15. 15.
    Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering 16(1), 61–102 (2011)CrossRefGoogle Scholar
  16. 16.
    Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies: a survey. Softw. Test, Verif. Reliab. 15(3), 167–199 (2005)CrossRefGoogle Scholar
  17. 17.
    Hadzic, T., Hansen, E.R.: On automata, MDDs and BDDs in constraint satisfaction. In: Proceedings of the ECAI 2008 Workshop on Inference Methods based on Graphical Structures of Knowledge (2008)Google Scholar
  18. 18.
    Kuhn, D.R., Reilly, M.J.: An investigation of the applicability of design of experiments to software testing. In: Society, I. (ed.) 27th NASA/IEEE Software Engineering Workshop, pp. 91–95 (2002)Google Scholar
  19. 19.
    Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Software Eng. 30(6), 418–421 (2004)CrossRefGoogle Scholar
  20. 20.
    Kuhn, R., Kacker, R., Lei, Y., Hunter, J.: Combinatorial software testing. Computer 42(8), 94–96 (2009)CrossRefGoogle Scholar
  21. 21.
    Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG/IPOG-D: efficient test generation for multi-way combinatorial testing. Software Testing, Verification and Reliability 18(3), 125–148 (2008)CrossRefGoogle Scholar
  22. 22.
    Nagayama, S., Sasao, T.: Compact representations of logic functions using heterogeneous MDDs. In: Proceedings of 33rd International Symposium on Multiple-Valued Logic, pp. 247–252 (2003)Google Scholar
  23. 23.
    Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), 11 (2011)CrossRefGoogle Scholar
  24. 24.
    Pairwise web site, http://www.pairwise.org/
  25. 25.
    Salecker, E., Reicherdt, R., Glesner, S.: Calculating prioritized interaction test sets with constraints using binary decision diagrams. In: Proceedings of IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, pp. 278–285. IEEE Computer Society (2011)Google Scholar
  26. 26.
    Segall, I., Tzoref-Brill, R., Farchi, E.: Using binary decision diagrams for combinatorial test design. In: Proceedings of the 2011 International Symposium on Software Testing and Analysis, ISSTA 2011, pp. 254–264. ACM, New York (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Angelo Gargantini
    • 1
  • Paolo Vavassori
    • 1
  1. 1.Dip. di IngegneriaUniversità di BergamoItaly

Personalised recommendations