Skip to main content
SpringerLink
Log in

Evaluating improvements to a meta-heuristic search for constrained interaction testing

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Combinatorial interaction testing (CIT) is a cost-effective sampling technique for discovering interaction faults in highly-configurable systems. Constrained CIT extends the technique to situations where some features cannot coexist in a configuration, and is therefore more applicable to real-world software. Recent work on greedy algorithms to build CIT samples now efficiently supports these feature constraints. But when testing a single system configuration is expensive, greedy techniques perform worse than meta-heuristic algorithms, because greedy algorithms generally need larger samples to exercise the same set of interactions. On the other hand, current meta-heuristic algorithms have long run times when feature constraints are present. Neither class of algorithm is suitable when both constraints and the cost of testing configurations are important factors. Therefore, we reformulate one meta-heuristic search algorithm for constructing CIT samples, simulated annealing, to more efficiently incorporate constraints. We identify a set of algorithmic changes and experiment with our modifications on 35 realistic constrained problems and on a set of unconstrained problems from the literature to isolate the factors that improve performance. Our evaluation determines that the optimizations reduce run time by a factor of 90 and accomplish the same coverage objectives with even fewer system configurations. Furthermore, the new version compares favorably with greedy algorithms on real-world problems, and, though our modifications were aimed at constrained problems, it shows similar advantages when feature constraints are absent.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Ordinary covering arrays, strictly speaking, are already constrained—they must cover all t-sets. However, in this paper we follow the conventions of the literature and use the terms constrained covering array and unconstrained covering array to refer to samples with and without feature constraints, respectively.

References

  • Bryce RC, Colbourn CJ (2006) Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inform Software Tech 48(10):960–970

    Article  Google Scholar 

  • Bryce R, Colbourn CJ (2007) One-test-at-a-time heuristic search for interaction test suites. In: 9th conference on genetic and evolutionary computation, pp 1082–1089

  • Bryce R, Colbourn CJ, Cohen MB (2005) A framework of greedy methods for constructing interaction tests. In: 27th international conference on software engineering, pp 146–155

  • Calvagna A, Gargantini A (2009) Combining satisfiability solving and heuristics to constrained combinatorial interaction testing. In: 3rd international conference on tests and proofs. Springer, pp 27–42

  • Chateauneuf M, Kreher DL (2002) On the state of strength-three covering arrays. J Comb Des 10(4):217–238

    Article  MATH  MathSciNet  Google Scholar 

  • Clements P, Northrup L (2002) Software product lines: practices and patterns. Addison-Wesley

  • Coello Coello C (2002) Theoretical and numerical constraint handling techniques used with evolutionary algorithms: a survey of the state of the art. Comput Methods Appl Mech Eng 191(11–12):1245–1287

    Article  MATH  MathSciNet  Google Scholar 

  • Cohen DM, Dalal SR, Fredman ML, Patton GC (1997) The AETG system: an approach to testing based on combinatorial design. IEEE Trans Softw Eng 23(7):437–444

    Article  Google Scholar 

  • Cohen MB, Colbourn CJ, Ling ACH (2003a) Augmenting simulated annealing to build interaction test suites. In: 14th international symposium on software reliability engineering, pp 394–405

  • Cohen MB, Gibbons PB, Mugridge WB, Colbourn CJ (2003b) Constructing test suites for interaction testing. In: 25th international conference on software engineering, pp 38–48

  • Cohen MB, Dwyer MB, Shi J (2007a) Exploiting constraint solving history to construct interaction test suites. In: 2nd testing: academic and industrial conference—practice and research techniques, pp 121–130

  • Cohen MB, Dwyer MB, Shi J (2007b) Interaction testing of highly-configurable systems in the presence of constraints. In: 5th international symposium on software testing and analysis, pp 129–139

  • Cohen MB, Dwyer MB, Shi J (2008) Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans Softw Eng 34(5):633–650

    Article  Google Scholar 

  • Colbourn C (2009) Covering array tables. Available at http://www.public.asu.edu/~ccolbou/src/tabby/catable.html. Accessed 28 September 2009

  • Colbourn CJ (2004) Combinatorial aspects of covering arrays. Le Matematiche (Catania) 58:121–167

    Google Scholar 

  • Colbourn CJ, Cohen MB, Turban RC (2004) A deterministic density algorithm for pairwise interaction coverage. In: 1st IASTED international conference on software engineering, pp 345–352

  • Czerwonka J (2006) Pairwise testing in real world. In: 10th Pacific northwest software quality conference, pp 419–430

  • Eén N, Sörrenson N (2007) MiniSAT-C v1.14.1. Available at http://minisat.se/. Accessed 29 June 2010

  • Fouché S, Cohen MB, Porter A (2009) Incremental covering array failure characterization in large configuration spaces. In: 7th international symposium on software testing and analysis, pp 177–187

  • Garousi V (2008) Empirical analysis of a genetic algorithm-based stress test technique. In: Proceedings of annual conference on genetic and evolutionary computation, pp 1743–1750

  • Garvin BJ, Cohen MB, Dwyer MB (2009) An improved meta-heuristic search for constrained interaction testing. In: 1st international symposium on search based software engineering, pp 13–22

  • Grieskamp W, Qu X, Wei X, Kicillof N, Cohen MB (2009) Interaction coverage meets path coverage by SMT constraint solving. In: Testing of communicating systems and international workshop on formal approaches to testing of software

  • Harman M (2007) The current state and future of search based software engineering. In: Future of software engineering, pp 342–357

  • Harman M, Tratt L (2007) Pareto optimal search based refactoring at the design level. In: 9th conference on genetic and evolutionary computation, pp 1106–1113

  • Harman M, Swift S, Mahdavi K (2005) An empirical study of the robustness of two module clustering fitness functions. In: 7th conference on genetic and evolutionary computation, pp 1029–1036

  • Hartman A, Raskin L (2004) Problems and algorithms for covering arrays. Discrete Math 284:149–156

    Article  MATH  MathSciNet  Google Scholar 

  • Hnich B, Prestwich S, Selensky E, Smith BM (2006) Constraint models for the covering test problem. Constraints 11:199–219

    Article  MATH  MathSciNet  Google Scholar 

  • Kuhn DR, Wallace DR, Gallo AM (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421

    Article  Google Scholar 

  • Lei Y, Tai KC (1998) In-parameter-order: a test generation strategy for pairwise testing. In: 3rd international symposium on high-assurance systems engineering, pp 254–261

  • Lei Y, Kacker R, Kuhn DR, Okun V, Lawrence J (2007) IPOG: a general strategy for t-way software testing. In: 14th international conference on the engineering of computer-based systems, pp 549–556

    Google Scholar 

  • Li Z, Harman M, Hierons RM (2007) Search algorithms for regression test case prioritization. IEEE Trans Softw Eng 33(4):225–237

    Article  Google Scholar 

  • Malik S (2004) zChaff. Available at http://www.princeton.edu/~chaff/zchaff.html. Accessed 29 June 2010

  • McMinn P (2004) Search-based software test data generation: a survey. Softw Test Verif Reliab 14(2):105–156

    Article  Google Scholar 

  • Memon A, Porter A, Yilmaz C, Nagarajan A, Schmidt DC, Natarajan B (2004) Skoll: distributed continuous quality assurance. In: 26th international conference on software engineering

  • Nurmela KJ (2004) Upper bounds for covering arrays by tabu search. Discrete Appl Math 138(1–2):143–152

    Article  MATH  MathSciNet  Google Scholar 

  • Pohl K, Böckle G, van der Linden F (2005) Software product line engineering. Springer, Berlin

    MATH  Google Scholar 

  • Qu X, Cohen MB, Rothermel G (2008) Configuration-aware regression testing: an empirical study of sampling and prioritization. In: 6th international symposium on software testing and analysis, pp 75–85

  • Stardom J (2001) Metaheuristics and the search for covering and packing arrays. Master’s thesis, Simon Fraser University

  • Stevens B (1998) Transversal covers and packings. PhD thesis, University of Toronto

  • Tai KC, Lei Y (2002) A test generation strategy for pairwise testing. IEEE Trans Softw Eng 28(1):109–111

    Article  Google Scholar 

  • Yilmaz C, Cohen MB, Porter A (2006) Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans Softw Eng 31(1):20–34

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank Jiangfan Shi for the use of his constrained mAETG tool and for supplying the CIT models for evaluation. Brady Garvin is supported in part by CFDA#84.200A: Graduate Assistance in Areas of National Need (GAANN). This work is supported in part by the National Science Foundation through awards CNS-0454203, CCF-0541263, CNS-0720654, and CCF-0747009, by the Air Force Office of Scientific Research through award FA9550-09-1-0129, the Army Research Office through DURIP award W91NF-04-1-0104, the Defense Advanced Research Projects Agency through award HR0011-09-1-0031 and through the National Aeronautics and Space Administration under grant number NNX08AV20A. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the position or policy of NSF, AFOSR, ARO, DARPA or NASA.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Nebraska—Lincoln, Lincoln, NE, 68588-0115, USA

    Brady J. Garvin, Myra B. Cohen & Matthew B. Dwyer

Authors
  1. Brady J. Garvin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Myra B. Cohen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthew B. Dwyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Myra B. Cohen.

Additional information

Editors: Massimiliano Di Penta and Simon Poulding

Rights and permissions

Reprints and permissions

About this article

Cite this article

Garvin, B.J., Cohen, M.B. & Dwyer, M.B. Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empir Software Eng 16, 61–102 (2011). https://doi.org/10.1007/s10664-010-9135-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-010-9135-7

Keywords

Search

Navigation