Optimization Letters

, Volume 11, Issue 3, pp 457–469 | Cite as

A two-stage algorithm for combinatorial testing

  • Jose Torres-Jimenez
  • Himer Avila-George
  • Idelfonso Izquierdo-Marquez
Original Paper


Covering arrays are combinatorial structures which have applications in fields like software testing and hardware Trojan detection. In this paper we proposed a two-stage simulated annealing algorithm to construct covering arrays. The proposed algorithm is instanced in this paper through the construction of ternary covering arrays of strength three. We were able to get 579 new upper bounds. In order to show the generality of our proposal, we defined a new benchmark composed of 25 instances of MCAs taken from the literature, all instances were improved.


Combinatorial testing Simulated annealing Covering arrays 



The authors acknowledge to ABACUS-CINVESTAV, CONACYT grant EDOMEX-2011-COI-165873 for providing access of high performance computing. The authors acknowledge General Coordination of Information and Communications Technologies (CGSTIC) at CINVESTAV for providing HPC resources on the Hybrid Cluster Supercomputer “Xiuhcoatl”, that have contributed to the research results reported. The following projects have funded the research reported in this paper: 148784—FOMIX—Unidad de Transferencia Tecnologica CICESE—Nayarit; 238469—CONACyT Métodos Exactos para Construir Covering Arrays Óptimos; 2143—Cátedras CONACyT—Fortalecimiento de las capacidades de TICs en Nayarit.


  1. 1.
    Ansótegui, C., Izquierdo-Marquez, I., Manya, F., Torres-Jimenez, J.: A Max-SAT-Based approach to constructing optimal covering arrays. Artificial intelligence research and development. CCIA 2013, 51–59 (2013)Google Scholar
  2. 2.
    Avila-George, H., Torres-Jimenez, J., Gonzalez-Hernandez, L., Hernández, V.: Metaheuristic approach for constructing functional test-suites. IET Softw. 7(2), 104–117 (2013)CrossRefGoogle Scholar
  3. 3.
    Avila-George, H., Torres-Jimenez, J., Hernández, V., Gonzalez-Hernandez, L.: Simulated annealing for constructing mixed covering arrays. In: Distributed computing and artificial intelligence, pp. 657–664 (2012)Google Scholar
  4. 4.
    Bush, K.A.: Orthogonal arrays of index unity. Ann. Math. Stat. 23(3), 426–434 (1952)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Calvagna, A., Gargantini, A.: T-wise combinatorial interaction test suites construction based on coverage inheritance. Softw. Test. Verif. Reliab. 22, 507–526 (2012)CrossRefGoogle Scholar
  6. 6.
    Cawse, J.N.: Experimental design for combinatorial and high throughput materials development. Wiley, New York (2003)Google Scholar
  7. 7.
    Chateauneuf, M., Kreher, D.L.: On the state of strength-three covering arrays. J. Comb. Des. 10(4), 217–238 (2002)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Cohen, D.M., Dalal, S.R., Parelius, J., Patton, G.C.: The combinatorial design approach to automatic test generation. IEEE Soft. 13(5), 83–88 (1996)CrossRefGoogle Scholar
  9. 9.
    Cohen, M.B., Colbourn, C.J., Ling, A.C.H.: Augmenting simulated annealing to build interaction test suites. In: Software reliability engineering, 2003. ISSRE 2003. 14th international symposium on IEEE, pp. 394–405. (2003)Google Scholar
  10. 10.
    Cohen, M.B., Colbourn, C.J., Ling, A.C.H.: Constructing strength three covering arrays with augmented annealing. Discret. Math. 308(13), 2709–2722 (2008). (Combinatorial Designs: A tribute to Jennifer Seberry on her 60th Birthday)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Colbourn, C.J.: Covering arrays from cyclotomy. Des. Codes Cryptogr. 55(2–3), 201–219 (2010)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Colbourn, C.J. (2015). Covering array tables. Accessed on 11 Mar 2015
  13. 13.
    Colbourn, C.J., Martirosyan, S.S., Trung, T., Walker II, R.A.: Roux-type constructions for covering arrays of strengths three and four. Des. Codes Cryptogr. 41(1), 33–57 (2006)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Duan, F., Lei, Y., Yu, L., Kacker, R.N., Kuhn, D.R.: Improving IPOG’s vertical growth based on a graph coloring scheme. In: 2015 IEEE Eighth international conference on software testing, verification and validation workshops (ICSTW), pp. 1–8 (2015)Google Scholar
  15. 15.
    Flottes, M.-L., Dupuis, S., Ba, P.-S., Rouzeyre, B.: On the limitations of logic testing for detecting hardware trojans horses. In: Design technology of integrated systems in nanoscale era (DTIS), 2015 10th international conference on, pp. 1–5 (2015)Google Scholar
  16. 16.
    Forbes, M., Lawrence, J., Lei, Y., Kacker, R.N., Kuhn, D.R.: Refining the in-parameter-order strategy for constructing covering arrays. J. Res. Natl. Inst. Stand. Technol. 113(5), 287–297 (2008)CrossRefGoogle Scholar
  17. 17.
    Hartman, A.: Software and hardware testing using combinatorial covering suites. In: Graph theory, combinatorics and algorithms, vol. 34 of operations research/computer science interfaces series, pp. 237–266 (2005)Google Scholar
  18. 18.
    Kitsos, P., Simos, D.E., Torres-Jimenez, J., Voyiatzis, A.G.: Exciting FPGA cryptographic trojans using combinatorial testing. In: The 26th IEEE international symposium on software reliability engineering (2015) (To appear)Google Scholar
  19. 19.
    Kuhn, D.R., Kacker, R.N., Lei, Y.: Practical combinatorial testing. Technical report, National Institute of Standards and Technology (2010)Google Scholar
  20. 20.
    Martinez-Pena, J., Torres-Jimenez, J.: A branch and bound algorithm for ternary covering arrays construction using trinomial coefficients. Res. Comput. Sci. 49, 61–71 (2010)Google Scholar
  21. 21.
    Meagher, K., Stevens, B.: Group construction of covering arrays. J. Comb. Des. 13(1), 70–77 (2005)MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    Shasha, D.E., Kouranov, A.Y., Lejay, L.V., Chou, M.F., Coruzzi, G.M.: Using combinatorial design to study regulation by multiple input signals: A tool for parsimony in the post-genomics era. Plant Physiol. 127(4), 1590–1594 (2001)CrossRefGoogle Scholar
  23. 23.
    Sherwood, G. (2015). On the construction of orthogonal arrays and covering arrays using permutation groups. Accessed 20 Mar 2015
  24. 24.
    Shiba, T., Tsuchiya, T., Kikuno, T.: Using artificial life techniques to generate test cases for combinatorial testing. In: Proceedings of the 28th annual international computer software and applications Conference, IEEE Computer Society, pp. 72–77 (2004)Google Scholar
  25. 25.
    Sloane, N.J.A.: Covering arrays and intersecting codes. J. Comb. Des. 1(1), 51–63 (1993)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Torres-Jimenez, J., Avila-George, H., Rangel-Valdez, N., Gonzalez-Hernandez, L.: Construction of orthogonal arrays of index unity using logarithm tables for galois fields, chapter 4. InTech, pp. 71–90 (2012)Google Scholar
  27. 27.
    Torres-Jimenez, J., Rodriguez-Tello, E.: New bounds for binary covering arrays using simulated annealing. Inf. Sci. 185(1), 137–152 (2012)CrossRefGoogle Scholar
  28. 28.
    Walker II, R.A., Colbourn, C.J.: Tabu search for covering arrays using permutation vectors. J. Stat. Plan. Inference 139(1), 69–80 (2009)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Younis, M.I., Zamli, K.Z.: MIPOG - An Efficient t-Way Minimization Strategy for Combinatorial Testing. Int. J. Comput. Theory Eng. 3(3), 388–397 (2011)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Jose Torres-Jimenez
    • 1
  • Himer Avila-George
    • 2
  • Idelfonso Izquierdo-Marquez
    • 1
  1. 1.Information Technology LaboratoryCINVESTAV-TamaulipasCiudad VictoriaMexico

Personalised recommendations