Covering Arrays: Using Prior Information for Construction, Evaluation and to Facilitate Fault Localization

  • Ryan LekivetzEmail author
  • Joseph Morgan
Original Article
Part of the following topical collections:
  1. Algorithms, Analysis and Advanced Methodologies in the Design of Experiments


Covering arrays are increasingly being used by test engineers to derive test cases to test complex engineered systems. This approach to testing is known as combinatorial testing and has proven to be a cost-efficient way to determine test cases that are highly effective at identifying faults in the system that are due to the combination of several inputs. However, when such faults are encountered and failures occur, the test engineer is tasked with determining the inputs and associated values that triggered the failures. This exercise typically involves examining a long list of potential causes and may even require performing follow-up tests to reduce the number of potential causes. This paper addresses this issue by considering the prior knowledge of the system under test that test engineers often have. We show how this knowledge can be used to evaluate and improve the effectiveness of a suite of test cases before any test cases are executed. Finally, we address the case where failures occur and show how this prior knowledge can aid in determining the inputs, and associated values, that triggered the failures. In addition, throughout the paper, we compare and contrast the use of covering arrays for testing complex engineered systems to the use of factorial experiments in traditional experimental design settings.


Covering arrays Design of experiments Fault localization Prior information 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Box G, Tyssedal J (1996) Projective properties of certain orthogonal arrays. Biometrika 83(4):950–955MathSciNetCrossRefGoogle Scholar
  2. 2.
    Bryce RC, Colbourn CJ (2006) Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inf Softw Technol 48(10):960–970. (Advances in model-based testing) CrossRefGoogle Scholar
  3. 3.
    Chen T, He T, Benesty M, Khotilovich V, Tang Y (2015) Xgboost: extreme gradient boosting. R package version 0.4-2, pp 1–4Google Scholar
  4. 4.
    Colbourn CJ, Martirosyan SS, Mullen GL, Shasha D, Sherwood GB, Yucas JL (2006) Products of mixed covering arrays of strength two. J Comb Des 14(2):124–138MathSciNetCrossRefGoogle Scholar
  5. 5.
    Colbourn CJ, McClary DW (2008) Locating and detecting arrays for interaction faults. J Comb Optim 15(1):17–48. MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Colbourn CJ, Syrotiuk VR (2016) Coverage, location, detection, and measurement. In: IEEE ninth international conference on software testing, verification and validation workshops (ICSTW), pp 19–25.
  7. 7.
    Dalal SR, Mallows CL (1998) Factor-covering designs for testing software. Technometrics 40(3):234–243. CrossRefGoogle Scholar
  8. 8.
    Elbaum S, Rothermel G, Kanduri S, Malishevsky AG (2004) Selecting a cost-effective test case prioritization technique. Softw Qual J 12(3):185–210. CrossRefGoogle Scholar
  9. 9.
    Ghandehari LS, Lei Y, Kung D, Kacker R, Kuhn R (2013) Fault localization based on failure-inducing combinations. In: IEEE 24th international symposium on software reliability engineering (ISSRE), pp 168–177.
  10. 10.
    Gonzalez-Hernandez L, Torres-Jiménez J, Rangel-Valdez N (2011) An exact approach to maximize the number of wild cards in a covering array. In: Batyrshin I, Sidorov G (eds) Advances in artificial intelligence. MICAI 2011. Lecture notes in computer science, vol 7094. Springer, Berlin, HeidelbergGoogle Scholar
  11. 11.
    Katona GO (1973) Two applications (for search theory and truth functions) of sperner type theorems. Period Math Hung 3(1–2):19–26MathSciNetCrossRefGoogle Scholar
  12. 12.
    Kim Y, Jang DH, Anderson-Cook CM (2017) Selecting the best wild card entries in a covering array. Qual Reliab Eng Int 33(7):1615–1627CrossRefGoogle Scholar
  13. 13.
    Kleitman DJ, Spencer J (1973) Families of k-independent sets. Discret Math 6(3):255–262MathSciNetCrossRefGoogle Scholar
  14. 14.
    Kuhn DR, Okum V (2006) Pseudo-exhaustive testing for software. In: 30th Annual IEEE/NASA software engineering workshop. IEEE, pp 153–158Google Scholar
  15. 15.
    Kuhn DR, Reilly MJ (2002) An investigation of the applicability of design of experiments to software testing. In: Proceedings 27th annual NASA Goddard/IEEE software engineering workshop. IEEE, pp 91–95Google Scholar
  16. 16.
    Kuhn DR, Wallace DR, Gallo AM (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421. CrossRefGoogle Scholar
  17. 17.
    Lekivetz R, Morgan J (2018) Evaluation and construction of covering arrays utilizing prior information. In: IEEE international symposium on software reliability engineering workshops (ISSREW). IEEE, pp 132–133Google Scholar
  18. 18.
    Lekivetz R, Morgan J (2018) Fault localization: analyzing covering arrays given prior information. In: IEEE international conference on software quality, reliability and security companion (QRS-C). IEEE, pp 116–121Google Scholar
  19. 19.
    Morgan J (2018) Combinatorial testing: an approach to systems and software testing based on covering arrays. In: Kenett RS, Ruggeri F, Faltin FW (eds) Analytic methods in systems and software testing. Wiley, pp 131–158Google Scholar
  20. 20.
    Qu X, Cohen MB, Rothermel G (2008) Configuration-aware regression testing: an empirical study of sampling and prioritization. In: Proceedings of the 2008 international symposium on software testing and analysis. ACM, pp 75–86Google Scholar
  21. 21.
    Rényi A (2007) Foundations of probability. Courier Corporation, North ChelmsfordzbMATHGoogle Scholar
  22. 22.
    Wu CJ, Hamada MS (2011) Experiments: planning, analysis, and optimization, vol 552. Wiley, HobokenzbMATHGoogle Scholar

Copyright information

© Grace Scientific Publishing 2019

Authors and Affiliations

  1. 1.JMP DivisionSAS InstituteCaryUSA

Personalised recommendations