Empirical Software Engineering

, Volume 16, Issue 1, pp 61–102 | Cite as

Evaluating improvements to a meta-heuristic search for constrained interaction testing

  • Brady J. Garvin
  • Myra B. Cohen
  • Matthew B. Dwyer


Combinatorial interaction testing (CIT) is a cost-effective sampling technique for discovering interaction faults in highly-configurable systems. Constrained CIT extends the technique to situations where some features cannot coexist in a configuration, and is therefore more applicable to real-world software. Recent work on greedy algorithms to build CIT samples now efficiently supports these feature constraints. But when testing a single system configuration is expensive, greedy techniques perform worse than meta-heuristic algorithms, because greedy algorithms generally need larger samples to exercise the same set of interactions. On the other hand, current meta-heuristic algorithms have long run times when feature constraints are present. Neither class of algorithm is suitable when both constraints and the cost of testing configurations are important factors. Therefore, we reformulate one meta-heuristic search algorithm for constructing CIT samples, simulated annealing, to more efficiently incorporate constraints. We identify a set of algorithmic changes and experiment with our modifications on 35 realistic constrained problems and on a set of unconstrained problems from the literature to isolate the factors that improve performance. Our evaluation determines that the optimizations reduce run time by a factor of 90 and accomplish the same coverage objectives with even fewer system configurations. Furthermore, the new version compares favorably with greedy algorithms on real-world problems, and, though our modifications were aimed at constrained problems, it shows similar advantages when feature constraints are absent.


Constrained combinatorial interaction testing Configurable software Search based software engineering 



We would like to thank Jiangfan Shi for the use of his constrained mAETG tool and for supplying the CIT models for evaluation. Brady Garvin is supported in part by CFDA#84.200A: Graduate Assistance in Areas of National Need (GAANN). This work is supported in part by the National Science Foundation through awards CNS-0454203, CCF-0541263, CNS-0720654, and CCF-0747009, by the Air Force Office of Scientific Research through award FA9550-09-1-0129, the Army Research Office through DURIP award W91NF-04-1-0104, the Defense Advanced Research Projects Agency through award HR0011-09-1-0031 and through the National Aeronautics and Space Administration under grant number NNX08AV20A. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the position or policy of NSF, AFOSR, ARO, DARPA or NASA.


  1. Bryce RC, Colbourn CJ (2006) Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inform Software Tech 48(10):960–970CrossRefGoogle Scholar
  2. Bryce R, Colbourn CJ (2007) One-test-at-a-time heuristic search for interaction test suites. In: 9th conference on genetic and evolutionary computation, pp 1082–1089Google Scholar
  3. Bryce R, Colbourn CJ, Cohen MB (2005) A framework of greedy methods for constructing interaction tests. In: 27th international conference on software engineering, pp 146–155Google Scholar
  4. Calvagna A, Gargantini A (2009) Combining satisfiability solving and heuristics to constrained combinatorial interaction testing. In: 3rd international conference on tests and proofs. Springer, pp 27–42Google Scholar
  5. Chateauneuf M, Kreher DL (2002) On the state of strength-three covering arrays. J Comb Des 10(4):217–238MATHCrossRefMathSciNetGoogle Scholar
  6. Clements P, Northrup L (2002) Software product lines: practices and patterns. Addison-WesleyGoogle Scholar
  7. Coello Coello C (2002) Theoretical and numerical constraint handling techniques used with evolutionary algorithms: a survey of the state of the art. Comput Methods Appl Mech Eng 191(11–12):1245–1287MATHCrossRefMathSciNetGoogle Scholar
  8. Cohen DM, Dalal SR, Fredman ML, Patton GC (1997) The AETG system: an approach to testing based on combinatorial design. IEEE Trans Softw Eng 23(7):437–444CrossRefGoogle Scholar
  9. Cohen MB, Colbourn CJ, Ling ACH (2003a) Augmenting simulated annealing to build interaction test suites. In: 14th international symposium on software reliability engineering, pp 394–405Google Scholar
  10. Cohen MB, Gibbons PB, Mugridge WB, Colbourn CJ (2003b) Constructing test suites for interaction testing. In: 25th international conference on software engineering, pp 38–48Google Scholar
  11. Cohen MB, Dwyer MB, Shi J (2007a) Exploiting constraint solving history to construct interaction test suites. In: 2nd testing: academic and industrial conference—practice and research techniques, pp 121–130Google Scholar
  12. Cohen MB, Dwyer MB, Shi J (2007b) Interaction testing of highly-configurable systems in the presence of constraints. In: 5th international symposium on software testing and analysis, pp 129–139Google Scholar
  13. Cohen MB, Dwyer MB, Shi J (2008) Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans Softw Eng 34(5):633–650CrossRefGoogle Scholar
  14. Colbourn C (2009) Covering array tables. Available at http://www.public.asu.edu/~ccolbou/src/tabby/catable.html. Accessed 28 September 2009
  15. Colbourn CJ (2004) Combinatorial aspects of covering arrays. Le Matematiche (Catania) 58:121–167Google Scholar
  16. Colbourn CJ, Cohen MB, Turban RC (2004) A deterministic density algorithm for pairwise interaction coverage. In: 1st IASTED international conference on software engineering, pp 345–352Google Scholar
  17. Czerwonka J (2006) Pairwise testing in real world. In: 10th Pacific northwest software quality conference, pp 419–430Google Scholar
  18. Eén N, Sörrenson N (2007) MiniSAT-C v1.14.1. Available at http://minisat.se/. Accessed 29 June 2010
  19. Fouché S, Cohen MB, Porter A (2009) Incremental covering array failure characterization in large configuration spaces. In: 7th international symposium on software testing and analysis, pp 177–187Google Scholar
  20. Garousi V (2008) Empirical analysis of a genetic algorithm-based stress test technique. In: Proceedings of annual conference on genetic and evolutionary computation, pp 1743–1750Google Scholar
  21. Garvin BJ, Cohen MB, Dwyer MB (2009) An improved meta-heuristic search for constrained interaction testing. In: 1st international symposium on search based software engineering, pp 13–22Google Scholar
  22. Grieskamp W, Qu X, Wei X, Kicillof N, Cohen MB (2009) Interaction coverage meets path coverage by SMT constraint solving. In: Testing of communicating systems and international workshop on formal approaches to testing of softwareGoogle Scholar
  23. Harman M (2007) The current state and future of search based software engineering. In: Future of software engineering, pp 342–357Google Scholar
  24. Harman M, Tratt L (2007) Pareto optimal search based refactoring at the design level. In: 9th conference on genetic and evolutionary computation, pp 1106–1113Google Scholar
  25. Harman M, Swift S, Mahdavi K (2005) An empirical study of the robustness of two module clustering fitness functions. In: 7th conference on genetic and evolutionary computation, pp 1029–1036Google Scholar
  26. Hartman A, Raskin L (2004) Problems and algorithms for covering arrays. Discrete Math 284:149–156MATHCrossRefMathSciNetGoogle Scholar
  27. Hnich B, Prestwich S, Selensky E, Smith BM (2006) Constraint models for the covering test problem. Constraints 11:199–219MATHCrossRefMathSciNetGoogle Scholar
  28. Kuhn DR, Wallace DR, Gallo AM (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421CrossRefGoogle Scholar
  29. Lei Y, Tai KC (1998) In-parameter-order: a test generation strategy for pairwise testing. In: 3rd international symposium on high-assurance systems engineering, pp 254–261Google Scholar
  30. Lei Y, Kacker R, Kuhn DR, Okun V, Lawrence J (2007) IPOG: a general strategy for t-way software testing. In: 14th international conference on the engineering of computer-based systems, pp 549–556Google Scholar
  31. Li Z, Harman M, Hierons RM (2007) Search algorithms for regression test case prioritization. IEEE Trans Softw Eng 33(4):225–237CrossRefGoogle Scholar
  32. Malik S (2004) zChaff. Available at http://www.princeton.edu/~chaff/zchaff.html. Accessed 29 June 2010
  33. McMinn P (2004) Search-based software test data generation: a survey. Softw Test Verif Reliab 14(2):105–156CrossRefGoogle Scholar
  34. Memon A, Porter A, Yilmaz C, Nagarajan A, Schmidt DC, Natarajan B (2004) Skoll: distributed continuous quality assurance. In: 26th international conference on software engineeringGoogle Scholar
  35. Nurmela KJ (2004) Upper bounds for covering arrays by tabu search. Discrete Appl Math 138(1–2):143–152MATHCrossRefMathSciNetGoogle Scholar
  36. Pohl K, Böckle G, van der Linden F (2005) Software product line engineering. Springer, BerlinMATHGoogle Scholar
  37. Qu X, Cohen MB, Rothermel G (2008) Configuration-aware regression testing: an empirical study of sampling and prioritization. In: 6th international symposium on software testing and analysis, pp 75–85Google Scholar
  38. Stardom J (2001) Metaheuristics and the search for covering and packing arrays. Master’s thesis, Simon Fraser UniversityGoogle Scholar
  39. Stevens B (1998) Transversal covers and packings. PhD thesis, University of TorontoGoogle Scholar
  40. Tai KC, Lei Y (2002) A test generation strategy for pairwise testing. IEEE Trans Softw Eng 28(1):109–111CrossRefGoogle Scholar
  41. Yilmaz C, Cohen MB, Porter A (2006) Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans Softw Eng 31(1):20–34CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Brady J. Garvin
    • 1
  • Myra B. Cohen
    • 1
  • Matthew B. Dwyer
    • 1
  1. 1.Department of Computer Science and EngineeringUniversity of Nebraska—LincolnLincolnUSA

Personalised recommendations