The Journal of Supercomputing

, Volume 62, Issue 2, pp 916–945 | Cite as

Supercomputing and grid computing on the verification of covering arrays

  • Himer Avila-George
  • Jose Torres-Jimenez
  • Nelson Rangel-Valdez
  • Abel Carrión
  • Vicente Hernández


The Covering Arrays (CAs) are mathematical objects with minimal coverage and maximum cardinality that are a good tool for the design of experiments. A covering array is an N×k matrix over an alphabet v s.t. each N×k subset contains at least one time each combination from {0,1,…,v−1} t , given a positive integer value t. The process of ensuring that a CA contains each of the v t combinations is called verification of CA. In this paper, we present an algorithm for CA verification and its implementation details in three different computation paradigms: (a) sequential approach (SA); (b) parallel approach (PA); and (c) Grid approach (GA). Four different PAs were compared in their performance of verifying a matrix as a CA; the PA with the best performance was included in a different experimentation where the three paradigms, SA, PA, and GA were compared in a benchmark composed by 45 possible CA instances. The results showed the limitations of the different paradigms when solving the verification of CA problem, and points out the necessity of a Grid approach to solve the problem when the size of a CA grows.


Covering array Combinatorial testing Supercomputing Grid computing 



The authors thankfully acknowledge the computer resources and assistance provided by Spanish Supercomputing Network (TIRANT-UV). This research work was partially funded by the following projects: CONACyT 58554, Calculo de Covering Arrays; 51623 Fondo Mixto CONACyT y Gobierno del Estado de Tamaulipas.


  1. 1.
    Avila-George H, Torres-Jimenez J, Hernández V, Rangel-Valdez N (2010) Verification of general and cyclic covering arrays using Grid computing. In: Proceedings of the 3rd international conference on data management in grid and peer-to-peer systems, GLOBE 2010, Bilbao, Spain, 30 August–3 September. Lecture notes in computer science, vol 6265. Springer, Berlin, pp 112–123. doi: 10.1007/978-3-642-15108-8_10 Google Scholar
  2. 2.
    Bryce RC, Colbourn CJ (2007) The density algorithm for pairwise interaction testing. Softw Test Verif Reliab 17(3):159–182. doi: 10.1002/stvr.365 CrossRefGoogle Scholar
  3. 3.
    Burr K, Young W (1998) Combinatorial test techniques: table-based automation, test generation and code coverage. In: Proceedings of the international conference on software testing, analysis, and review—STAR, pp 503–513. West, 1998 Google Scholar
  4. 4.
    Bush KA (1952) Orthogonal arrays of index unity. Ann Math Stat 23(3):426–434. doi: 10.1214/aoms/1177729387 MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Calvagna A, Gargantini A, Tramontana E (2009) Building T-wise combinatorial interaction test suites by means of grid computing. In: Proceedings of the 18th IEEE international workshops on enabling technologies: infrastructures for collaborative enterprises—WETICE 2009. IEEE Comput Soc, Los Alamitos, pp 213–218. doi: 10.1109/WETICE.2009.52 CrossRefGoogle Scholar
  6. 6.
    Cawse JN (2003) Experimental design for combinatorial and high throughput materials development. Wiley, New York Google Scholar
  7. 7.
    Cheng C (2007) The test suite generation problem: optimal instances and their implications. Discrete Appl Math 155:1943–1957. doi: 10.1016/j.dam.2007.04.010 MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Cohen DM, Dalal SR, Parelius J, Patton GC (1996) The combinatorial design approach to automatic test generation. IEEE Softw 13(5):83–88. doi: 10.1109/52.536462 CrossRefGoogle Scholar
  9. 9.
    Cohen MB, Colbourn CJ, Ling ACH (2003) Augmenting simulated annealing to build interaction test suites. In: Proceedings of the 14th international symposium on software reliability engineering—ISSRE 2003. IEEE Comput Soc, Los Alamitos, pp 394–405. CrossRefGoogle Scholar
  10. 10.
    Colbourn CJ (2004) Combinatorial aspects of covering arrays. Matematiche 59(1, 2):125–172 MathSciNetzbMATHGoogle Scholar
  11. 11.
    Colbourn CJ (2011) Covering array tables for t=2,3,4,5,6. URL: Accessed on April 20
  12. 12.
    Colbourn CJ, Kéri G (2009) Binary covering arrays and existentially closed graphs. In: Proceedings of the 2nd international workshop on coding and cryptology—IWCC 2009. Lecture notes in computer science, vol 5557. Springer, Berlin, pp 22–33. doi: 10.1007/978-3-642-01877-0_3 Google Scholar
  13. 13.
    Colbourn CJ, Ling ACH (2009) A recursive construction for perfect hash families. J Math Cryptol 3(4):291–306. doi: 10.1515/JMC.2009.018 MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Colbourn CJ, Martirosyan SS, Mullen GL, Shasha D, Sherwood GB, Yucas JL (2006) Products of mixed covering arrays of strength two. J Comb Des 12(2):124–138. doi: 10.1002/jcd.20065 MathSciNetCrossRefGoogle Scholar
  15. 15.
    Colbourn CJ, Torres-Jimenez J (2010) Heterogeneous hash families and covering arrays. In: Error-correcting codes, finite geometries and cryptography. Contemporary mathematics, vol 523. pp 3–15. ISBN-10 0-8218-4956-5 CrossRefGoogle Scholar
  16. 16.
    DIANE (2011) Distributed analysis environment. URL: Accessed on June 6
  17. 17.
    Foster I, Kesselman C (1999) The grid: blueprint for a new computing infrastructure. Morgan Kaufmann, San Mateo Google Scholar
  18. 18.
    Seroussi NBG (1988) Vector sets for exhaustive testing of logic circuits. IEEE Trans Inf Theory 34:513–522 MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Gonzalez-Hernandez L, Rangel-Valdez N, Torres-Jimenez J (2010) Construction of mixed covering arrays of variable strength using a tabu search approach. In: Proceedings of the 4th international conference on combinatorial optimization and applications, COCOA, 2010. Lecture notes in computer science, vol 6508. Springer, Berlin, pp 51–64. doi: 10.1007/978-3-642-17458-2_6 CrossRefGoogle Scholar
  20. 20.
    Hedayat AS, Sloane NJA, Stufken J (1999) Orthogonal arrays: theory and applications. Springer, Berlin zbMATHGoogle Scholar
  21. 21.
    Katona GOH (1973) Two applications (for search theory and truth functions) of Sperner type theorems. Period Math Hung 3(1–2):19–26. doi: 10.1007/BF02018457 MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Kleitman DJ, Spencer J (1973) Families of k-independent sets. Discrete Math 6(3):255–262. doi: 10.1016/0012-365X(73)90098-8 MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Kuhn R, Lei Y, Kacker R (2008) Practical combinatorial testing: beyond pairwise. IT Prof 10(3):19–23. doi: 10.1109/MITP.2008.54 CrossRefGoogle Scholar
  24. 24.
    Lawrence J, Kacker R, Lei Y, Kuhn D, Forbes M (2011) A survey of binary covering arrays. Electron J Comb 18(1):84 MathSciNetGoogle Scholar
  25. 25.
    Lei Y, Kacker R, Kuhn DR, Okun V, Lawrence J (2007) IPOG: a general strategy for t-way software testing. In: Proceedings of the 14th annual IEEE international conference and workshops on the engineering of computer-based systems—ECBS 2007. IEEE Comput Soc, Los Alamitos, pp 549–556. doi: 10.1109/ECBS.2007.47 Google Scholar
  26. 26.
    Martinez-Pena J, Torres-Jimenez J, Rangel-Valdez N, Avila-George H (2010) A heuristic approach for constructing ternary covering arrays using trinomial coefficients. In: Proceedings of the 12th Ibero-American conference on artificial intelligence—IBERAMIA 2010. Lecture notes in computer science, vol 6433. Springer, Berlin, pp 572–581. doi: 10.1007/978-3-642-16952-6_58 CrossRefGoogle Scholar
  27. 27.
    Martirosyan SS, Colbourn CJ (2005) Recursive constructions of covering arrays. Bayreuth Math Schr 74:266–275 MathSciNetzbMATHGoogle Scholar
  28. 28.
    McDowell AG (2011) All-pairs testing. URL: Accessed on June 21
  29. 29.
    Moscicki J, Brochu F, Ebke J, Egede U, Elmsheuser J, Harrison K, Jones R, Lee H, Liko D, Maier A, Muraru A, Patrick G, Pajchel K, Reece W, Samset B, Slater M, Soroko A, Tan C, van der Ster D, Williams M (2009) Ganga: a tool for computational-task management and easy access to grid resources. Comput Phys Commun 180(11):2303–2316. doi: 10.1016/j.cpc.2009.06.016 CrossRefGoogle Scholar
  30. 30.
    Moura L, Stardom J, Stevens B, Williams A (2003) Covering arrays with mixed alphabet sizes. J Comb Des 11(6):413–432. doi: 10.1002/jcd.10059 MathSciNetzbMATHGoogle Scholar
  31. 31.
    National Institute of Standards and Technology (2011) NIST covering array tables. URL: Accessed on April 20
  32. 32.
    Nurmela KJ (2004) Upper bounds for covering arrays by tabu search. Discrete Appl Math 138:143–152. doi: 10.1016/S0166-218X(03)00291-9 MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Pacini F (2011) Job description language howto. URL: Accessed on October 10
  34. 34.
    Phadke MS (1995) Quality engineering using robust design. Prentice Hall, New York Google Scholar
  35. 35.
    Rényi A (1971) Foundations of probability. Wiley, New York Google Scholar
  36. 36.
    Shasha DE, Kouranov AY, Lejay LV, Chou MF, Coruzzi GM (2001) Using combinatorial design to study regulation by multiple input signals: a tool for parsimony in the post-genomics era. Plant Physiol 127(4):1590–1594. doi: 10.1104/pp.010683 CrossRefGoogle Scholar
  37. 37.
    Sherwood GB (2008) Optimal and near-optimal mixed covering arrays by column expansion. Discrete Math 308(24):6022–6035. doi: 10.1016/j.disc.2007.11.021 MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    Sherwood GB (2011) On the construction of orthogonal arrays and covering arrays using permutation groups. URL: Accessed on June 20
  39. 39.
    Shiba T, Tsuchiya T, Kikuno T (2004) Using artificial life techniques to generate test cases for combinatorial testing. In: Proceedings of the 28th annual international computer software and applications conference, vol 01, COMPSAC 2004. IEEE Comput Soc, Los Alamitos, pp 72–77. doi: 10.1109/CMPSAC.2004.1342808 CrossRefGoogle Scholar
  40. 40.
    Torres-Jimenez J, Avila-George H, Rangel-Valdez N, Gonzalez-Hernandez L (2012) Construction of orthogonal arrays of index unity using logarithm tables for galois fields. In Cryptography. InTech, pp 71–90. ISBN 978-953-51-0179-6 Google Scholar
  41. 41.
    Torres-Jimenez J, De Alfonso C, Hernández V (2004) Computation of ternary covering arrays using a grid. In: Proceedings of the second Asian applied computing conference—AACC 2004. Lecture notes in computer science, vol 3285. Springer, Berlin, pp 240–246. doi: 10.1007/978-3-540-30176-9_31 Google Scholar
  42. 42.
    Torres-Jimenez J, Rangel-Valdez N, Gonzalez-Hernandez AL, Avila-George H (2010) Construction of logarithm tables for Galois fields. Int J Math Educ Sci Technol 42(1):91–102. doi: 10.1080/0020739X.2010.510215 MathSciNetCrossRefGoogle Scholar
  43. 43.
    Torres-Jimenez J, Rodriguez-Tello E (2012) New bounds for binary covering arrays using simulated annealing. Inf Sci 185(1):137–152. doi: 10.1016/j.ins.2011.09.020 CrossRefGoogle Scholar
  44. 44.
    Tung Y, Aldiwan WS (2000) Automating test case generation for the new generation mission software system. In: Proceedings of the IEEE aerospace conference, vol 1. IEEE Press, New York, pp 431–437. doi: 10.1109/AERO.2000.879426 Google Scholar
  45. 45.
    Vadde K, Syrotiuk V (2004) Factor interaction on service delivery in mobile ad hoc networks. IEEE J Sel Areas Commun 22(7):1335–1346. doi: 10.1109/JSAC.2004.829351 CrossRefGoogle Scholar
  46. 46.
    Williams AW (2000) Determination of test configurations for pair-wise interaction coverage. In: Proceedings of the IFIP TC6/WG6.1 13th International conference on testing communicating systems: tools and techniques—TestCom. Kluwer, Norwell, pp 59–74 Google Scholar
  47. 47.
    Williams AW, Probert RL (1996) A practical strategy for testing pair-wise coverage of network interfaces. In: Proceedings of the seventh international symposium on software reliability engineering—ISSRE. IEEE Comput Soc, Los Alamitos, pp 246–256. doi: 10.1109/ISSRE.1996.558835 CrossRefGoogle Scholar
  48. 48.
    Yilmaz C, Cohen MB, Porter AA (2006) Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans Softw Eng 32(1):20–34. doi: 10.1109/TSE.2006.8 CrossRefGoogle Scholar
  49. 49.
    Younis M, Zamli K, Isa N (2008) A strategy for grid based t-way test data generation. In: First international conference on distributed framework and applications—DFmA 2008, pp 73–78. doi: 10.1109/ICDFMA.2008.4784416 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Himer Avila-George
    • 1
  • Jose Torres-Jimenez
    • 2
  • Nelson Rangel-Valdez
    • 3
  • Abel Carrión
    • 1
  • Vicente Hernández
    • 1
  1. 1.Instituto de Instrumentación para Imagen Molecular (I3M), Centro mixto CSICUniversitat Politècnica de València, CIEMATValenciaSpain
  2. 2.Information Technology LaboratoryCINVESTAV-TamaulipasVictoria TampsMexico
  3. 3.Universidad Politécnica de Ciudad VictoriaParque Científico y Tecnológico de TamaulipasCd. Victoria, Tamps.Mexico

Personalised recommendations