Set-Based Algorithms for Combinatorial Test Set Generation

  • Ludwig Kampel
  • Dimitris E. Simos
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9976)


Testing is an important and expensive part of software and hardware development. Over the recent years, the construction of combinatorial interaction tests rose to play an important role towards making the cost of testing more efficient. Covering arrays are the key element of combinatorial interaction testing and a means to provide abstract test sets. In this paper, we present a family of set-based algorithms for generating covering arrays and thus combinatorial test sets. Our algorithms build upon an existing mathematical method for constructing independent families of sets, which we extend sufficiently in terms of algorithmic design in this paper. We compare our algorithms against commonly used greedy methods for producing 3-way combinatorial test sets, and these initial evaluation results favor our approach in terms of generating smaller test sets.


Combinatorial testing Independent families of sets Set-based algorithms 



This work has been funded by the Austrian Research Promotion Agency (FFG) under grant 851205 and the Austrian COMET Program (FFG).


  1. 1.
    The economic impacts of inadequate infrastructure for software testing. U.S. Department of Commerce, National Institute of Standards and Technology (2002)Google Scholar
  2. 2.
    Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23(7), 437–444 (1997)CrossRefGoogle Scholar
  3. 3.
    Colbourn, C.J.: Table for CAN(3, k, 2) for k up to 10000. Accessed 25 Apr 2016
  4. 4.
    Colbourn, C.J.: Combinatorial aspects of covering arrays. Le Matematiche (Catania) 58, 121–167 (2004)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Colbourn, C.J., Dinitz, J.H.: Handbook of Combinatorial Designs. CRC Press, Boca Raton (2006)CrossRefzbMATHGoogle Scholar
  6. 6.
    Forbes, M., Lawrence, J., Lei, Y., Kacker, R., Kuhn, D.R.: Refining the in-parameter-order strategy for constructing covering arrays. J. Res. Nat. Inst. Stand. Technol. 113, 287–297 (2008)CrossRefGoogle Scholar
  7. 7.
    Freiman, G., Lipkin, E., Levitin, L.: A polynomial algorithm for constructing families of k-independent sets. Discret. Math. 70(2), 137–147 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Gao, S.W., Lv, J.H., Du, B.L., Colbourn, C.J., Ma, S.L.: Balancing frequencies and fault detection in the in-parameter-order algorithm. J. Comput. Sci. Technol. 30(5), 957–968 (2015)CrossRefGoogle Scholar
  9. 9.
    Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. Discret. Math. 284(13), 149–156 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Kleitman, D.J., Spencer, J.: Families of k-independent sets. Discret. Math. 6(3), 255–262 (1973)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Kuhn, D., Kacker, R., Lei, Y.: Practical combinatorial testing. In: NIST Special Publication pp. 800–142 (2010)Google Scholar
  12. 12.
    Kuhn, D., Kacker, R., Lei, Y.: Introduction to Combinatorial Testing. Chapman & Hall/CRC Innovations in Software Engineering and Software Development Series. Taylor & Francis, New York (2013)zbMATHGoogle Scholar
  13. 13.
    Lawrence, J., Kacker, R.N., Lei, Y., Kuhn, D.R., Forbes, M.: A survey of binary covering arrays. Electron. J. Comb. 18(1), P84 (2011)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing. Softw. Test. Verif. Reliab. 18(3), 125–148 (2008)CrossRefGoogle Scholar
  15. 15.
    Lei, Y., Tai, K.C.: In-parameter-order: a test generation strategy for pairwise testing. In: 1998 3rd IEEE International Proceedings of High-Assurance Systems Engineering Symposium, pp. 254–261. IEEE (1998)Google Scholar
  16. 16.
    NIST: Table for CA(3, k, 2). National Institute of Standards and Technology. Accessed 25 Apr 2016
  17. 17.
    Seroussi, G., Bshouty, N.H.: Vector sets for exhaustive testing of logic circuits. IEEE Trans. Inf. Theor. 34(3), 513–522 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Torres-Jimenez, J., Izquierdo-Marquez, I.: Survey of covering arrays. In: 2013 15th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pp. 20–27. IEEE (2013)Google Scholar
  19. 19.
    Yu, L., Lei, Y., Kacker, R.N., Kuhn, D.R.: Acts: a combinatorial test generation tool. In: 2013 IEEE 6th International Conference on Software Testing, Verification and Validation (ICST), pp. 370–375. IEEE (2013)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2016

Authors and Affiliations

  1. 1.SBA ResearchViennaAustria

Personalised recommendations