catnap: Generating Test Suites of Constrained Combinatorial Testing with Answer Set Programming

  • Mutsunori Banbara
  • Katsumi Inoue
  • Hiromasa Kaneyuki
  • Tenda Okimoto
  • Torsten SchaubEmail author
  • Takehide Soh
  • Naoyuki Tamura
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10377)


We develop an approach to test suite generation for Constrained Combinatorial Testing (CCT), one of the most widely studied combinatorial testing techniques, based on Answer Set Programming (ASP). The resulting catnap system accepts a CCT instance in fact format and combines it with a first-order encoding for generating test suites, which can subsequently be solved by any off-the-shelf ASP systems. We evaluate the effectiveness of our approach by empirically contrasting it to the best known bounds obtained via dedicated implementations.


  1. 1.
    Balduccini, M.: Representing constraint satisfaction problems in answer set programming. In: Faber, W., Lee, J. (eds.) Proceedings of the Second Workshop on Answer Set Programming and Other Computing Paradigms (ASPOCP 2009), pp. 16–30 (2009)Google Scholar
  2. 2.
    Banbara, M., Inoue, K., Kaufmann, B., Schaub, T., Soh, T., Tamura, N., Wanko, P.: teaspoon: Solving the curriculum-based course timetabling problems with answer set programming. In: Proceedings of the Eleventh International Conference of the Practice and Theory of Automated Timetabling (PATAT 2016), pp. 13–32 (2016)Google Scholar
  3. 3.
    Banbara, M., Tamura, N., Inoue, K.: Generating event-sequence test cases by answer set programming with the incidence matrix. In: Technical Communications of the 28th International Conference on Logic Programming (ICLP 2012), LIPIcs, vol. 17, pp. 86–97. Schloss Dagstuhl (2012)Google Scholar
  4. 4.
    Baral, C.: Knowledge Representation, Reasoning and Declarative Problem Solving. Cambridge University Press, Cambridge (2003)CrossRefzbMATHGoogle Scholar
  5. 5.
    Brain, M., Erdem, E., Inoue, K., Oetsch, J., Pührer, J., Tompits, H., Yilmaz, C.: Event-sequence testing using answer-set programming. Int. J. Adv. Softw. 5(3–4), 237–251 (2012)Google Scholar
  6. 6.
    Bryce, R.C., Colbourn, C.J.: Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inf. Softw. Technol. 48(10), 960–970 (2006)CrossRefGoogle Scholar
  7. 7.
    Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatiorial design. IEEE Trans. Softw. Eng. 23(7), 437–444 (1997)CrossRefGoogle Scholar
  8. 8.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34(5), 633–650 (2008)CrossRefGoogle Scholar
  9. 9.
    Colbourn, C.J., Dinitz, J.H.: Handbook of Combinatorial Designs, 2nd edn. Chapman & Hall/CRC, Boca Raton (2006)CrossRefzbMATHGoogle Scholar
  10. 10.
    Czerwonka, J.: Pairwise testing in real world. Practical extensions to test case generators. In: Proceedings of the 24th Annual Pacific Northwest Software Quality Conference, pp. 419–430 (2006)Google Scholar
  11. 11.
    Erdem, E., Inoue, K., Oetsch, J., Pührer, J., Tompits, H., Yilmaz, C.: Answer-set programming as a new approach to event-sequence testing. In: Proceedings of the 3rd International Conference on Advances in System Testing and Validation Lifecycle (VALID 2011), pp. 25–34. Xpert Publishing Services (2011)Google Scholar
  12. 12.
    Garvin, B.J., Cohen, M.B., Dwyer, M.B.: An improved meta-heuristic search for constrained interaction testing. In: Proceedings of the first International Symposium on Search Based Software Engineering (SSBSE 2009), pp. 13–22. IEEE (2009)Google Scholar
  13. 13.
    Gebser, M., Ostrowski, M., Schaub, T.: Constraint answer set solving. In: Hill, P.M., Warren, D.S. (eds.) ICLP 2009. LNCS, vol. 5649, pp. 235–249. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-02846-5_22 CrossRefGoogle Scholar
  14. 14.
    Lin, J., Luo, C., Cai, S., Su, K., Hao, D., Zhang, L.: TCA: An efficient two-mode meta-heuristic algorithm for combinatorial test generation. In: Proceedings of 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015), pp. 494–505. IEEE (2015)Google Scholar
  15. 15.
    Mandl, R.: Orthogonal latin squares: an application of experiment design to compiler testing. Commun. ACM 28(10), 1054–1058 (1985)CrossRefGoogle Scholar
  16. 16.
    Nanba, T., Tsuchiya, T., Kikuno, T.: Constructing test sets for pairwise testing: a SAT-based approach. In: Proceedings of the Second International Conference on Networking and Computing (ICNC 2011), pp. 271–274. IEEE (2011)Google Scholar
  17. 17.
    Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), 11:1–11:29 (2011). CrossRefzbMATHGoogle Scholar
  18. 18.
    Petke, J., Yoo, S., Cohen, M.B., Harman, M.: Efficiency and early fault detection with lower and higher strength combinatorial interaction testing. In: Proceedings of ESEC/FSE 2013, pp. 26–36. ACM (2013)Google Scholar
  19. 19.
    Tatsumi, K.: Test case design support system. In: Proceedings of the International Conference on Quality Control (ICQC 1987), pp. 615–620 (1987)Google Scholar
  20. 20.
    Yamada, A., Kitamura, T., Artho, C., Choi, E., Oiwa, Y., Biere, A.: Optimization of combinatorial testing by incremental SAT solving. In: Proceedings of 8th IEEE International Conference on Software Testing, Verification and Validation (ICST 2015), pp. 1–10. IEEE (2015)Google Scholar
  21. 21.
    Yu, L., Lei, Y., Nourozborazjany, M., Kacker, R., Kuhn, D.R.: An efficient algorithm for constraint handling in combinatorial test generation. In: Proceedings of the Sixth IEEE International Conference on Software Testing, Verification and Validation (ICST 2013), pp. 242–251. IEEE (2013)Google Scholar
  22. 22.
    Zhang, Z., Yan, J., Zhao, Y., Zhang, J.: Generating combinatorial test suite using combinatorial optimization. J. Syst. Softw. 98, 191–207 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Mutsunori Banbara
    • 1
  • Katsumi Inoue
    • 2
  • Hiromasa Kaneyuki
    • 1
  • Tenda Okimoto
    • 1
  • Torsten Schaub
    • 3
    Email author
  • Takehide Soh
    • 1
  • Naoyuki Tamura
    • 1
  1. 1.Kobe UniversityKobeJapan
  2. 2.NIIChiyoda-kuJapan
  3. 3.Universität PotsdamPotsdamGermany

Personalised recommendations