Generating Cost-Aware Covering Arrays for Free

  • Mustafa Kemal TaşEmail author
  • Hanefi Mercan
  • Gülşen Demiröz
  • Kamer Kaya
  • Cemal Yilmaz
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 779)


Software systems generally have a large number of configurable options interacting with each other. Such systems are more likely to be prone to errors, crashes, and faulty executions that are usually caused by option interactions. To avoid such errors, testing all possible configurations during the development phase is usually not feasible, since the number of all possible configurations is exponential in the order of number of options. A t-way covering array (CA) is a 2-dimensional combinatorial object that helps to efficiently cover all t-length option interactions of the system under test. Generating a CA with a small number of configurations is important to shorten the testing phase. However, the testing cost (e.g. the testing time) may differ from one configuration to another. Currently, most sequential tools can generate optimum CAs in terms of number of configurations, but they are not cost-aware, i.e., they cannot handle the varying costs of configurations. In this work, we implement a parallel, cost-aware CA-generation tool based on a sequential tool, Jenny, to generate lower-cost CAs faster. Experimental results show that our cost-aware CA construction approach can generate \(32\%\) and \(21\%\) lower cost CAs on average for t = 2 and t = 3, respectively, compared to state-of-the-art CA-generation tools. Moreover, the cost-awareness comes for free, i.e., we speed up our algorithm by leveraging parallel computation. The cost models and cost reduction techniques we propose could also be adapted for other existing CA generation tools.


Software testing Testing cost Combinatorial interaction testing Covering arrays Cost-aware testing Parallel covering array generation 


  1. 1.
    Kuhn, D.R., Wallace, D.R., Gallo Jr., A.M.: Software fault interactions and implications for software testing. IEEE Trans. Software Eng. 30(6), 418–421 (2004)CrossRefGoogle Scholar
  2. 2.
    Yilmaz, C., Fouche, S., Cohen, M.B., Porter, A., Demiroz, G., Koc, U.: Moving forward with combinatorial interaction testing. Computer 2, 37–45 (2014)CrossRefGoogle Scholar
  3. 3.
    Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. (CSUR) 43(2), 11 (2011)CrossRefzbMATHGoogle Scholar
  4. 4.
    Seroussi, G., Bshouty, N.H.: Vector sets for exhaustive testing of logic circuits. IEEE Trans. Inf. Theory 34(3), 513–522 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Lei, Y., Tai, K.-C.: In-parameter-order: a test generation strategy for pairwise testing. In: High-Assurance Systems Engineering Symposium, Proceedings Third IEEE International, pp. 254–261. IEEE (1998)Google Scholar
  6. 6.
    Cohen, M.B., Colbourn, C.J., Ling, A.C.: Augmenting simulated annealing to build interaction test suites. In: 14th International Symposium on Software Reliability Engineering, ISSRE 2003, pp. 394–405. IEEE (2003)Google Scholar
  7. 7.
    Czerwonka, J.: Pairwise testing in the real world: practical extensions to test-case scenarios. In: Proceedings of 24th Pacific Northwest Software Quality Conference, pp. 419–430. Citeseer (2006)Google Scholar
  8. 8.
    Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: Ipog/ipog-d: efficient test generation for multi-way combinatorial testing. Softw. Test. Verification Reliab. 18(3), 125–148 (2008)CrossRefGoogle Scholar
  9. 9.
    Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Softw. Eng. 16(1), 61–102 (2011)CrossRefGoogle Scholar
  10. 10.
    Khalsa, S.K., Labiche, Y.: An orchestrated survey of available algorithms and tools for combinatorial testing. In: 2014 IEEE 25th International Symposium on Software Reliability Engineering (ISSRE), pp. 323–334. IEEE (2014)Google Scholar
  11. 11.
    Demiroz, G., Yilmaz, C.: Cost-aware combinatorial interaction testing. In: In the Proceedings of VALID 2012, The Fourth International Conference on Advances in System Testing and Validation Lifecycle, pp. 9–16, November 2012Google Scholar
  12. 12.
    Demiroz, G., Yilmaz, C.: Using simulated annealing for computing cost-aware covering arrays. Appl. Soft Comput. 49, 1129–1144 (2016)CrossRefGoogle Scholar
  13. 13.
    Demiroz, G.: Cost-aware combinatorial interaction testing (doctoral symposium). In: Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015), pp. 440–443. ACM, July 2015Google Scholar
  14. 14.
    Bryce, R.C., Colbourn, C.J.: Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inf. Softw. Technol. 48(10), 960–970 (2006). Advances in Model-based TestingCrossRefGoogle Scholar
  15. 15.
    Jenkins, B.: jenny: A pairwise testing tool (2005).
  16. 16.
    Pairwise testing available tools.
  17. 17.
    Johansen, M.F., Haugen, O., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. In: Proceedings of the 16th International Software Product Line Conference, SPLC 2012, vol. 1, pp. 46–55. ACM (2012)Google Scholar
  18. 18.
    Demiroz, G., Yilmaz, C.: Towards automatic cost model discovery for combinatorial interaction testing. In: Proceedings of the 2016 International Workshop on Combinatorial Testing (IWCT 2016). IEEE, April 2016Google Scholar
  19. 19.
    Yu, L., Lei, Y., Kacker, R.N., Kuhn, D.R.: Acts: a combinatorial test generation tool. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), pp. 370–375. IEEE (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Faculty of Engineering and Natural SciencesSabancı UniversityIstanbulTurkey

Personalised recommendations