Logging to Facilitate Combinatorial System Testing

  • Peter M. Kruse
  • I. S. Wishnu B. Prasetya
  • Jurriaan Hage
  • Alexander Elyasov
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8432)


Testing a web application is typically very complicated. Imposing simple coverage criteria such as function or line coverage is often not sufficient to uncover bugs due to incorrect components integration. Combinatorial testing can enforce a stronger criterion, while still allowing the prioritization of test cases in order to keep the overall effort feasible. Combinatorial testing requires the whole testing domain to be classified and formalized, e.g., in terms of classification trees. At the system testing level, these trees can be quite large. This short paper presents our preliminary work to automatically construct classification trees from loggings of the system, and to subsequently calculate the coverage of our test runs against various combinatorial criteria. We use the tool CTE which allows such criteria to be custom specified. Furthermore, it comes with a graphical interface to simplify the specification of new test sequences.


Combinatorial testing Classification trees Logging 



This work is supported by EU grant ICT-257574 (FITTEST).


  1. 1.
    Memon, A.M., Soffa, M.L., Pollack, M.E.: Coverage criteria for GUI testing. In: Proceedings of the 8th European Software Engineering Conference Held Jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ESEC/FSE-9, pp. 256–267. ACM, New York (2001)Google Scholar
  2. 2.
    Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43, 11:1–11:29 (2011)CrossRefGoogle Scholar
  3. 3.
    Grochtmann, M., Grimm, K.: Classification trees for partition testing. Softw. Test. Verif. Reliab. 3(2), 63–82 (1993)CrossRefGoogle Scholar
  4. 4.
    Musa, J.D.: Operational profiles in software-reliability engineering. IEEE Softw. 10(2), 14–32 (1993)CrossRefGoogle Scholar
  5. 5.
    Misra, R.B., Saravana Kumar, K.: Software operational profile based test case allocation using fuzzy logic. Int. J. Autom. Comput. 4(4), 388 (2007)CrossRefGoogle Scholar
  6. 6.
    Bach, J.: Exploratory testing explained (2003). http://www.satisfice.com/articles/et-article.pdf
  7. 7.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: ISSTA ’07: Proceedings of the 2007 International Symposium on Software Testing and Analysis, New York, NY, USA, pp. 129–139 (2007)Google Scholar
  8. 8.
    Lehmann, E., Wegener, J.: Test case design by means of the CTE XL. In: Proceedings of the 8th European International Conference on Software Testing, Analysis and Review (EuroSTAR 2000), Kopenhagen, Denmark, December 2000Google Scholar
  9. 9.
    Kruse, P.M., Luniak, M.: Automated test case generation using classification trees. Softw. Qual. Prof. 13(1), 4–12 (2010)Google Scholar
  10. 10.
    Walton, G.H., Poore, J.H., Trammell, C.J.: Statistical testing of software based on a usage model. Softw. Pract. Exper. 25(1), 97–108 (1995)CrossRefGoogle Scholar
  11. 11.
    Elbaum, S., Malishevsky, A.G., Rothermel, G.: Test case prioritization: a family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002)CrossRefGoogle Scholar
  12. 12.
    Amland, S.: Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study. J. Syst. Softw. 53(3), 287–295 (2000)CrossRefGoogle Scholar
  13. 13.
    Kruse, P.M., Wegener, J.: Test sequence generation from classification trees. In: Proceedings of ICST 2012 Workshops (ICSTW 2012), Montreal, Canada (2012)Google Scholar
  14. 14.
    Conrad, M., Drr, H., Fey, I., Yap, A.: Model-based generation and structured representation of test scenarios. In: Proceedings of the Workshop on Software-Embedded Systems Testing, Gaithersburg, Maryland, USA (1999)Google Scholar
  15. 15.
    Prasetya, I.S.W.B., Middelkoop, A., Elyasov, A., Hage, J.: D6.1: FITTEST Logging Approach, Project no. 257574, FITTEST Future Internet Testing (2011)Google Scholar
  16. 16.
    Tonella, P., Marchetto, A., Nguyen, D.C., Jia, Y., Lakhotia, K., Harman, M.: Finding the optimal balance between over and under approximation of models inferred from execution logs. In: 2012 IEEE 5th International Conference on Software Testing, Verification and Validation, pp. 21–30. IEEE (2012)Google Scholar
  17. 17.
    Singh, H., Conrad, M., Sadeghipour, S.: Test case design based on Z and the classification-tree method. In: Proceedings of the 1st International Conference on Formal Engineering Methods, ICFEM ’97, pp. 81–90. IEEE Computer Society, Washington, DC (1997)Google Scholar
  18. 18.
    Hierons, R.M., Harman, M.: Automatically generating information from a Z specification to support the classification tree method. In: Bert, D., Bowen, J.P., King, S., Waldén, M. (eds.) ZB 2003. LNCS, vol. 2651, pp. 388–407. Springer, Heidelberg (2003) CrossRefGoogle Scholar
  19. 19.
    Feather, M.S.: Rapid application of lightweight formal methods for consistency analyses. IEEE Trans. Softw. Eng. 24(11), 949–959 (1998)CrossRefGoogle Scholar
  20. 20.
    Ducasse, S., Girba, T., Wuyts, R.: Object-oriented legacy system trace-based logic testing. In: 10th European Conference on Software Maintenance and Reengineering (CSMR). IEEE (2006)Google Scholar
  21. 21.
    Marchetto, A., Tonella, P., Ricca, F.: State-based testing of Ajax web applications. In: ICST, pp. 121–130. IEEE (2008)Google Scholar
  22. 22.
    Lorenzoli, D., Mariani, L., Pezzè, M.: Automatic generation of software behavioral models. In: 30th International Conference on Software Engineering, pp. 501–510. ACM (2008)Google Scholar
  23. 23.
    Nguyen, C.D., Marchetto, A., Tonella, P.: Combining model-based and combinatorial testing for effective test case generation. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA), Minneapolis, Minnesota, USA (2012)Google Scholar
  24. 24.
    Jain, S., Singh, I., Chandra, A., Zhang, Z.-L., Bronevetsky, G.: Extracting the textual and temporal structure of supercomputing logs. In: Yang, Y., Parashar, M., Muralidhar, R., Prasanna, V.K. (eds.) HiPC, pp. 254–263. IEEE (2009)Google Scholar
  25. 25.
    Ernst, M.D., Perkins, J.H., Guo, P.J., McCamant, S., Pacheco, C., Tschantz, M.S., Xiao, C.: The Daikon system for dynamic detection of likely invariants. Sci. Comput. Program. 69(1–3), 35–45 (2007)CrossRefMATHMathSciNetGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Peter M. Kruse
    • 1
  • I. S. Wishnu B. Prasetya
    • 2
  • Jurriaan Hage
    • 2
  • Alexander Elyasov
    • 2
  1. 1.Berner and Mattner Systemtechnik GmbHBerlinGermany
  2. 2.Department of Information and Computing SciencesUtrecht UniversityUtrechtThe Netherlands

Personalised recommendations