Abstract
Testing a web application is typically very complicated. Imposing simple coverage criteria such as function or line coverage is often not sufficient to uncover bugs due to incorrect components integration. Combinatorial testing can enforce a stronger criterion, while still allowing the prioritization of test cases in order to keep the overall effort feasible. Combinatorial testing requires the whole testing domain to be classified and formalized, e.g., in terms of classification trees. At the system testing level, these trees can be quite large. This short paper presents our preliminary work to automatically construct classification trees from loggings of the system, and to subsequently calculate the coverage of our test runs against various combinatorial criteria. We use the tool CTE which allows such criteria to be custom specified. Furthermore, it comes with a graphical interface to simplify the specification of new test sequences.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Memon, A.M., Soffa, M.L., Pollack, M.E.: Coverage criteria for GUI testing. In: Proceedings of the 8th European Software Engineering Conference Held Jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ESEC/FSE-9, pp. 256–267. ACM, New York (2001)
Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43, 11:1–11:29 (2011)
Grochtmann, M., Grimm, K.: Classification trees for partition testing. Softw. Test. Verif. Reliab. 3(2), 63–82 (1993)
Musa, J.D.: Operational profiles in software-reliability engineering. IEEE Softw. 10(2), 14–32 (1993)
Misra, R.B., Saravana Kumar, K.: Software operational profile based test case allocation using fuzzy logic. Int. J. Autom. Comput. 4(4), 388 (2007)
Bach, J.: Exploratory testing explained (2003). http://www.satisfice.com/articles/et-article.pdf
Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: ISSTA ’07: Proceedings of the 2007 International Symposium on Software Testing and Analysis, New York, NY, USA, pp. 129–139 (2007)
Lehmann, E., Wegener, J.: Test case design by means of the CTE XL. In: Proceedings of the 8th European International Conference on Software Testing, Analysis and Review (EuroSTAR 2000), Kopenhagen, Denmark, December 2000
Kruse, P.M., Luniak, M.: Automated test case generation using classification trees. Softw. Qual. Prof. 13(1), 4–12 (2010)
Walton, G.H., Poore, J.H., Trammell, C.J.: Statistical testing of software based on a usage model. Softw. Pract. Exper. 25(1), 97–108 (1995)
Elbaum, S., Malishevsky, A.G., Rothermel, G.: Test case prioritization: a family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002)
Amland, S.: Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study. J. Syst. Softw. 53(3), 287–295 (2000)
Kruse, P.M., Wegener, J.: Test sequence generation from classification trees. In: Proceedings of ICST 2012 Workshops (ICSTW 2012), Montreal, Canada (2012)
Conrad, M., Drr, H., Fey, I., Yap, A.: Model-based generation and structured representation of test scenarios. In: Proceedings of the Workshop on Software-Embedded Systems Testing, Gaithersburg, Maryland, USA (1999)
Prasetya, I.S.W.B., Middelkoop, A., Elyasov, A., Hage, J.: D6.1: FITTEST Logging Approach, Project no. 257574, FITTEST Future Internet Testing (2011)
Tonella, P., Marchetto, A., Nguyen, D.C., Jia, Y., Lakhotia, K., Harman, M.: Finding the optimal balance between over and under approximation of models inferred from execution logs. In: 2012 IEEE 5th International Conference on Software Testing, Verification and Validation, pp. 21–30. IEEE (2012)
Singh, H., Conrad, M., Sadeghipour, S.: Test case design based on Z and the classification-tree method. In: Proceedings of the 1st International Conference on Formal Engineering Methods, ICFEM ’97, pp. 81–90. IEEE Computer Society, Washington, DC (1997)
Hierons, R.M., Harman, M.: Automatically generating information from a Z specification to support the classification tree method. In: Bert, D., Bowen, J.P., King, S., Waldén, M. (eds.) ZB 2003. LNCS, vol. 2651, pp. 388–407. Springer, Heidelberg (2003)
Feather, M.S.: Rapid application of lightweight formal methods for consistency analyses. IEEE Trans. Softw. Eng. 24(11), 949–959 (1998)
Ducasse, S., Girba, T., Wuyts, R.: Object-oriented legacy system trace-based logic testing. In: 10th European Conference on Software Maintenance and Reengineering (CSMR). IEEE (2006)
Marchetto, A., Tonella, P., Ricca, F.: State-based testing of Ajax web applications. In: ICST, pp. 121–130. IEEE (2008)
Lorenzoli, D., Mariani, L., Pezzè, M.: Automatic generation of software behavioral models. In: 30th International Conference on Software Engineering, pp. 501–510. ACM (2008)
Nguyen, C.D., Marchetto, A., Tonella, P.: Combining model-based and combinatorial testing for effective test case generation. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA), Minneapolis, Minnesota, USA (2012)
Jain, S., Singh, I., Chandra, A., Zhang, Z.-L., Bronevetsky, G.: Extracting the textual and temporal structure of supercomputing logs. In: Yang, Y., Parashar, M., Muralidhar, R., Prasanna, V.K. (eds.) HiPC, pp. 254–263. IEEE (2009)
Ernst, M.D., Perkins, J.H., Guo, P.J., McCamant, S., Pacheco, C., Tschantz, M.S., Xiao, C.: The Daikon system for dynamic detection of likely invariants. Sci. Comput. Program. 69(1–3), 35–45 (2007)
Acknowledgment
This work is supported by EU grant ICT-257574 (FITTEST).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Kruse, P.M., Prasetya, I.S.W.B., Hage, J., Elyasov, A. (2014). Logging to Facilitate Combinatorial System Testing. In: Vos, T., Lakhotia, K., Bauersfeld, S. (eds) Future Internet Testing. FITTEST 2013. Lecture Notes in Computer Science(), vol 8432. Springer, Cham. https://doi.org/10.1007/978-3-319-07785-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-07785-7_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07784-0
Online ISBN: 978-3-319-07785-7
eBook Packages: Computer ScienceComputer Science (R0)