Skip to main content
SpringerLink
Log in

A survey of methods for constructing covering arrays

  • Published:
Programming and Computer Software Aims and scope Submit manuscript

Abstract

The paper presents a survey of methods for constructing covering arrays used in generation of tests for interfaces with a great number of parameters. The application domain of these methods and algorithms used in them are analyzed. Specific characteristics of the methods, including time complexity and estimates of the required memory, are presented. Various—direct, recursive, optimization, genetic, and backtracking—algorithms used for constructing covering arrays are presented. Heuristics are presented that allow one to reduce arrays without loss of completeness, and application domains of these heuristics are outlined.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Zelenov, S.V. and Zelenova, S.A., Automatic Generation of Positive and Negative Tests for Testing Syntactic Analysis Phase, Trudy Instituta Sistemnogo Programmirovaniya RAN (Proceedings of the Institute of System Programming, RAS), 2004, vol. 8, no. 1, pp. 41–58.

    Google Scholar 

  2. Hoffman, D., Sobotkiewicz, L., Wang Hong-Yi, Strooper, P., Bazdell, G., and Stevens, B., Test Generation with Context Free Grammars and Covering Arrays, in Testing: Academic and Industrial Conf. — Practice and Research Techniques, Windsor, UK, 2009, pp. 83–87.

  3. Ammann, P. and Offutt, J., Using Formal Methods to Derive Test Frames in Category-Partition Testing, Safety, Reliability, Fault Tolerance, Concurrency, and Real Time Security: Proc. of the 9-th Ann. Conf. on Computer Assurance (COMPASS’94), 1994, pp. 69–79.

  4. Cohen, D.M., Dalal, S.R., Fredman, M.L., and Patton, G.C., The AETG System: An Approach to Testing Based on Combinatorial Design, IEEE Trans. Software Engineering, 1996, vol. 23, no. 7, pp. 437–444.

    Article  Google Scholar 

  5. Williams, A.W., Determination of Test Configurations for Pairwise Interaction Coverage, Proc. of 13-th Int. Conf. on Testing Communicating Systems (TestCom), 2000, pp. 59–74.

  6. Williams, W. and Probert, R.L., A Measure for Component Interaction Test Coverage, Proc. ACS/IEEE Int. Conf. on Computer Systems and Applications, 2001, pp. 301–311.

  7. Bryce, R. and Colbourn, C.J., One-Test-at-a-Time Heuristic Search for Interaction Test Suites, Proc. of Genetic and Evolutionary Computation Conf. (GECCO), Search-based Software Engineering track (SBSE), London, 2007, pp. 1082–1089.

  8. Calvagna, A. and Gargantini, A., Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing, Proc. of the 3-rd Int. Conf. on Tests and Proofs, Zurich, 2009, pp. 27–42.

  9. Documentation on Service WM Keeper Light. http://webmoney.ru/rus/about/demo/light/index.shtml

  10. Patton, G.C., DAT (Defect Analysis Team) 1986–1990 Overview, Internal Bellcore Technical Memo, 1991.

  11. Cohen, D.M., Dalal, S.R., Parelius, J., and Patton, G.C., The Combinatorial Design Approach to Automatic Test Generation, IEEE Software, 1996, vol. 13, no. 5, pp. 83–87.

    Article  Google Scholar 

  12. Seroussi, G. and Bshouty, N.H., Vector Sets for Exhaustive Testing of Logic Circuits, IEEE Trans. Information Theory, 1988, vol. 34, no. 3, pp. 513–522.

    Article  MathSciNet  MATH  Google Scholar 

  13. Lei, Y. and Tai, K.C., In-parameter Order: A Test Generation Strategy for Pairwise Testing, Proc. of the 3rd IEEE High Assurance System Engineering Symp., 1998, pp. 254–261.

  14. Godbole, A.P., Skipper, D.E., and Sunley, R.A., T-Covering Arrays: Upper Bounds and Poisson Approximations, Combinatorics, Probability Computing, 1996, vol. 5, pp.105–118.

    Article  MathSciNet  MATH  Google Scholar 

  15. Martirosyan, S. and Van Trung, T., On T-covering Arrays, Designs, Codes and Cryptography, 2004, vol. 32, nos. 1–3, pp. 323–339.

    Article  MathSciNet  MATH  Google Scholar 

  16. Hartman, A. and Raskin, L., Problems and Algorithms for Covering Arrays, Discrete Math., 2004, vol. 284, pp. 149–156.

    Article  MathSciNet  MATH  Google Scholar 

  17. Colbourn, C.J., Combinatorial Aspects of Covering Arrays, Le Matematiche (Catania), 2004, vol. 58, pp. 121–167.

    Google Scholar 

  18. Grindal, M., Offutt, A.J., and Andler, S.F., Combination Testing Strategies: A Survey, Software Testing, Verification, Reliability, 2005, vol. 15, no. 3, pp. 167–199.

    Article  Google Scholar 

  19. Hartman, A., Software and Hardware Testing Using Combinatorial Covering Suites, Proc. of Graph Theory, Combinatorics and Algorithms: Interdisciplinary Applications, 2005, pp. 266–327.

  20. Colbourn, C.J., Keri, G., Rivas Soriano, P.P., Schlage-Puchta, J.-C., Covering and Radius-Covering Arrays: Constructions and Classification, Discrete Applied Math., 2010, vol. 158, no. 11, pp. 1158–1180.

    Article  MATH  Google Scholar 

  21. Colbourn, C.J., Strength Two Covering Arrays: Existence Tables and Projection, Discrete Math., 2008, vol. 308, nos. 5–6, pp. 772–786.

    Article  MathSciNet  MATH  Google Scholar 

  22. Forbes, M., Lawrence, J., Lei, Y., Kacker, R.N., and Kuhn, D.R., Refining the In-parameter-order Strategy for Constructing Covering Arrays, J. Res. Nat. Inst. Stand. Tech., 2008, vol. 113, no. 5, pp. 287–297.

    Google Scholar 

  23. Konstantinov, A.V., Automation of Test Construction with the Use of Combinatorial Methods, MS Thesis, Moscow State University, 2007.

  24. Cohen, M.B., Dwyer, M.B., and Shi, J., Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach, IEEE Trans. Software Eng., 2008, vol. 34, no, 5, pp. 633–650.

    Article  Google Scholar 

  25. Cohen, M.B., Dwyer, M.B., and Shi, J., Exploiting Constraint Solving History to Construct Interaction Test Suites, Testing: Academic and Industrial Conference Practice and Research Techniques — MUTATION, 2007, pp. 121–132.

  26. Czerwonka, J., Pairwise Testing in Real World, 24-th Pacific Northwest Software Quality Conference, 2006.

  27. Yan, J. and Zhang, J., A Backtracking Search Tool for Constructing Combinatorial Test Suites, J. System Software, 2008, vol. 81, no. 10, pp. 1681–1693.

    Article  MathSciNet  Google Scholar 

  28. Nayeri, P., Colbourn, C.J., and Konjevod, G., Randomized Postoptimization of Covering Arrays, Lecture Notes in Computer Science, Springer, 2009, vol. 5874, pp. 408–419.

    Article  MathSciNet  Google Scholar 

  29. Edelman, A., The Mathematics of the Pentium Division Bug, SIAM Review, 1997, vol. 39, no. 1, pp. 54–67.

    Article  MathSciNet  MATH  Google Scholar 

  30. Greene, C., Sperner Families and Partitions of a Partially Ordered Set, Combinatorics, Hall, Jr., M. and van Lint, J., Eds., Dordrecht, Holland, 1975, pp. 277–290.

  31. Lidl, R. and Niederreiter, H., Finite Fields, Reading, Mass.: Addison-Wesley, 1983. Translated under the title Konechnye polya, Moscow: Mir, 1988.

    MATH  Google Scholar 

  32. Chateauneuf, M. and Kreher, D., On the State of Strength-three Covering Arrays, J. Combinatorial Designs, 2002, vol. 10, no. 4, pp. 217–238.

    Article  MathSciNet  MATH  Google Scholar 

  33. Colbourn, J., Martirosyan, S.S., Mullen, G.L., Shasha, D., Yucas, J.L., and Sherwood, G.B., Products of Mixed Covering Arrays of Strength Two, J. Combinatorial Design, 2006, vol. 14, no. 2, pp. 124–138.

    Article  MathSciNet  MATH  Google Scholar 

  34. Meagher, K. and Stevens, B., Group Construction of Covering Arrays, J. Combinatorial Design, 2005, vol. 13, no. 1, pp. 70–77.

    Article  MathSciNet  MATH  Google Scholar 

  35. Yin, J., Constructions of Difference Covering Arrays, J. Combinatorial Theory (A), 2003, vol. 104, no. 2, pp. 327–339.

    Article  MATH  Google Scholar 

  36. Harari, F., Graph Theory, Reading, Mass.: Addison-Wesley, 1969. Translated under the title Teoriya grafov, Moscow: Mir, 1973.

    Google Scholar 

  37. Tables of the Known Least Covering Arrays. http://www.public.asu.edu/~ccolbou/src/tabby/catable.html.

  38. Sloane, N., Covering Arrays and Intersecting Codes, J. Combinatorial Designs, 1993, vol. 1, no. 1, pp. 51–63.

    Article  MathSciNet  MATH  Google Scholar 

  39. Cohen, M.B., Colbourn, C.J., and Ling, A.C.H., Constructing Strength Three Covering Arrays with Augmented Annealing, Discrete Math., 2008, vol. 308, no. 13, pp. 2709–2722.

    Article  MathSciNet  MATH  Google Scholar 

  40. Colbourn, C.J., Martirosyan, S.S., Van Trung, T., and Walker II, R.A., Roux-type Constructions for Covering Arrays of Strengths Three and Four, Designs, Codes Cryptography, 2006, vol. 41, no. 1, pp. 33–57.

    Article  MATH  Google Scholar 

  41. Stevens, B. and Mendelsohn, E., New Recursive Methods for Transversal Covers, J. Combinatorial Designs, 1999, vol. 7, no. 3, pp. 185–203.

    Article  MathSciNet  MATH  Google Scholar 

  42. Stevens, B., Ling, A., and Mendelsohn, E., A Direct Construction of Transversal Covers Using Group Divisible Designs, Ars Combin., 2002, vol. 63, pp. 145–159.

    MathSciNet  MATH  Google Scholar 

  43. Cormen, T., Leiserson, C., Rivest, R., and Stein, K., Introduction to Algorithms, The MIT Press, 2001, 2nd ed. Chapter 16: Greedy Algorithms.

  44. Calvaga, A. and Gargantini, A., IPO-s: Incremental Generation of Combinatorial Interaction Test Data Based on Symmetries of Covering Arrays, Proc. of the IEEE Int. Conf. on Software Testing Verification and Validation Workshops, Denver, Colorado, 2009.

  45. Cohen, M., Dalal, S.R., Fredman, M.L., and Patton, G.C., The Automatic Efficient Test Generator (AETG) System, Proc. of the 5-th Int. Symp. on Software Reliability Engineering, 1994, pp. 303–309.

  46. Colbourn, C.J., Cohen, M.B., and Turban, R.C., A Deterministic Density Algorithm for Pairwise Interaction Coverage, Proc. of the IASTED Int. Conf. on Software Engineering, 2004, pp. 242–252.

  47. Tung, Y.-W. and Aldiwan, W.S., Automating Test Case Generation for the New Generation Mission Software System, Proc. IEEE Aerospace Conf., 2000, pp. 431–437.

  48. Jenkins, B., Tool for Pairwise Testing. http://burtleburtle.net/bob/math/jenny.html, 2005.

  49. Pairwise Testing, Combinatorial Test Case Generation. http://www.pairwise.org/tools.asp.

  50. Nurmela, K., Upper Bunds for Covering Arrays by Tabu Search, Discrete Applied Math., 2004, vol. 138, nos. 1–2, pp. 143–152.

    Article  MathSciNet  MATH  Google Scholar 

  51. Stardom, J., Metaheuristic and the Search for Covering and Packing Arrays, MS Thesis, Simon Fraser University, 2001.

  52. Stevens, B., Transversal Covers and Packings, Ph. D. Thesis, Mathematics, Univ. of Toronto, 1998.

  53. Cohen, M.B., Colbourn, C.J., and Ling, A.C.H., Augmenting Simulated Annealing to Build Interaction Test Suites, Proc. of Int. Symp. Software Requirements Engineering, (ISSRE 2003), 2003, pp. 394–405.

  54. Dueck, G., New Optimization Heuristic — The Great Deluge Algorithm and the Record-To-Record Travel, J. Computational Phys., 1993, vol. 104, pp. 86–92.

    Article  MATH  Google Scholar 

  55. Dueck, G. and Scheuer, T., Threshold Accepting: A General Purpose Optimization Algorithm Appearing Superior to Simulating Annealing, J. Computational Phys., 1990, vol. 90, pp. 161–175.

    Article  MathSciNet  MATH  Google Scholar 

  56. Rodriguez-Tello, E. and Torres-Jimenez, J., Memetic Algorithms for Constructing Binary Covering Arrays of Strength Three, Proc. of Artificial Evolution 2009, pp. 86–97.

Download references

Author information

Authors and Affiliations

  1. Institute for System Programming, Russian Academy of Sciences, ul. Solzhenitsyna 25, Moscow, 109004, Russia

    V. V. Kuliamin & A. A. Petukhov

Authors
  1. V. V. Kuliamin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. A. Petukhov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to V. V. Kuliamin.

Additional information

Original Russian Text © V.V. Kuliamin, A.A. Petukhov, 2011, published in Programmirovanie, 2011, Vol. 37, No. 3.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kuliamin, V.V., Petukhov, A.A. A survey of methods for constructing covering arrays. Program Comput Soft 37, 121–146 (2011). https://doi.org/10.1134/S0361768811030029

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0361768811030029

Keywords

Search

Navigation