Innovations in Systems and Software Engineering

, Volume 12, Issue 4, pp 249–261 | Cite as

Measuring and specifying combinatorial coverage of test input configurations

  • D. Richard Kuhn
  • Raghu N. Kacker
  • Yu Lei
Original Paper


A key issue in testing is how many tests are needed for a required level of coverage or fault detection. Estimates are often based on error rates in initial testing, or on code coverage. For example, tests may be run until a desired level of statement or branch coverage is achieved. Combinatorial methods present an opportunity for a different approach to estimating required test set size, using characteristics of the test set. This paper describes methods for estimating the coverage of, and ability to detect, t-way interaction faults of a test set based on a covering array. We also develop a connection between (static) combinatorial coverage and (dynamic) code coverage, such that if a specific condition is satisfied, 100 % branch coverage is assured. Using these results, we propose practical recommendations for using combinatorial coverage in specifying test requirements, and for improving estimates of the fault detection capacity of a test set.


Combinatorial testing Configuration model Covering array Fault coverage State-space coverage T-way testing Verification and validation (V&V) 


  1. 1.
    Dunietz S, Ehrlich WK, Szablak BD, Mallows CL, Iannino A (1997) Applying design of experiments to software testing. In: Proc. Intl. Conf. on Software Engineering, (ICSE ’97), pp 205–215Google Scholar
  2. 2.
    Grindal M, Offutt J, Andler SF (2005) Combination testing strategies: a survey. Softw Test Verif Reliab 15:167–199CrossRefGoogle Scholar
  3. 3.
    Lei Y, Kacker R, Kuhn DR, Okun V, Lawrence J (2007) IPOG: a general strategy for \(t\)-way software testing. In: Proc., IEEE Engineering of computer based systems, pp 549–556Google Scholar
  4. 4.
    Kuhn DR, Kacker RN, Lei Y (2010) Practical combinatorial testing. NIST SP 800–142Google Scholar
  5. 5.
    Kuhn DR, Wallace DR, Gallo AM Jr (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421CrossRefGoogle Scholar
  6. 6.
    Bryce R, Colbourn CJ, Cohen MB (2005) A framework of greedy methods for constructing interaction tests. In: The 27th international conference on software engineering (ICSE), St. Louis, Missouri, pp 146–155Google Scholar
  7. 7.
    Kuhn DR, Kacker R, Lei Y (2012) Combinatorial coverage measurement. NIST IR 7878Google Scholar
  8. 8.
    Kuhn DR, Dominguez Mendoza I, Kacker RN, Lei Y (2013) Combinatorial coverage measurement concepts and applications. In: Proc. IEEE sixth intl conf on software testing, verification and validation workshops (IWCT). IEEE, pp 352–361Google Scholar
  9. 9.
    Maximoff JR, Trela MD, Kuhn DR, Kacker R (2010) A method for analyzing system state-space coverage within a \(t\)-wise testing framework. IEEE International Systems Conference 2010. Apr. 4–11, San DiegoGoogle Scholar
  10. 10.
    Kuhn DR, Kacker RN (2012) Measuring combinatorial coverage of system state-space for IV&V. NASA IV&V WorkshopGoogle Scholar
  11. 11.
    Price C, Kuhn R, Forquer R, Lagoy A, Kacker R (2013) Evaluating the \(t\)-way combinatorial technique for determining the thoroughness of a test suite. NASA IV&V WorkshopGoogle Scholar
  12. 12.
    Arcuri A, Briand L (2011) Formal analysis of the probability of interaction fault detection using random testing. IEEE Trans. Software Engineering. IEEE Computer SocietyGoogle Scholar
  13. 13.
    Lyu MR (1996) Handbook of software reliability engineering, vol 222. IEEE Computer Society Press, CAGoogle Scholar
  14. 14.
    Bell KZ, Vouk Mladen A (2005) On effectiveness of pairwise methodology for testing network-centric software. In: Proceedings of the ITI third IEEE intl conf. information & communications technology. Cairo, Egypt, 2005, pp 221–235Google Scholar
  15. 15.
    du Bousquet L, Ledru Y, Maury O, Oriat C, Lanet J-L (2004) A case study in JML-based software validation. In: Proc. 19th Int. IEEE Conf. on automated sofware engineering. Linz, 2004, pp 294–297Google Scholar
  16. 16.
    Ammann P, Offutt J (2008) Introduction to software testing. Cambridge University Press, Cambridge, UKCrossRefzbMATHGoogle Scholar
  17. 17.
    Montanez C, Kuhn DR, Brady M, Rivello RM, Reyes J, Michael K (2012) Evaluation of fault detection effectiveness for combinatorial and exhaustive selection of discretized test inputs. Softw Qual Prof Mag 14(3):32–38Google Scholar

Copyright information

© Springer-Verlag London (outside the USA) 2015

Authors and Affiliations

  1. 1.National Institute of Standards and TechnologyGaithersburgUSA
  2. 2.Computer Science and EngineeringUniv. of Texas ArlingtonArlingtonUSA

Personalised recommendations