Ariadne: Evolving Test Data Using Grammatical Evolution
Software testing is a key component in software quality assurance; it typically involves generating test data that exercises all instructions and tested conditions in a program and, due to its complexity, can consume as much as 50% of overall software development budget. Some evolutionary computing techniques have been successfully applied to automate the process of test data generation but no existing techniques exploit variable interdependencies in the process of test data generation, even though several studies from the software testing literature suggest that the variables examined in the branching conditions of real life programs are often interdependent on each other, for example, if (x == y), etc.
We propose the Ariadne system which uses Grammatical Evolution (GE) and a simple Attribute Grammar to exploit the variable interdependencies in the process of test data generation. Our results show that Ariadne dramatically improves both effectiveness and efficiency when compared with existing techniques based upon well-established criteria, attaining coverage (the standard software testing success metric for these sorts of problems) of 100% on all benchmarks with far fewer program evaluations (often between a third and a tenth of other systems).
KeywordsAutomatic test case generation Code coverage Evolutionary testing Grammatical Evolution Variable interdependencies
The authors would like to thank Muhammad Hamad Khan for his help with the graphic designs. This work is supported by Lero, the Irish Software Research Centre, and the Science Foundation of Ireland.
- 1.Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing. Wiley, Hoboken (2011)Google Scholar
- 3.Sauder, R.L.: A general test data generator for COBOL. In: Proceedings of the 1–3 May 1962, Spring Joint Computer Conference, pp. 317–323. ACM (1962)Google Scholar
- 11.Cohen, E.I.: A finite domain-testing strategy for computer program testing. Ph.D. thesis, The Ohio State University (1978)Google Scholar
- 15.Harman, M., Jia, Y., Zhang, Y.: Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–12. IEEE (2015)Google Scholar
- 25.Korel, B.: Automated test data generation for programs with procedures. In: ACM SIGSOFT Software Engineering Notes, vol. 21, pp. 209–215. ACM (1996)Google Scholar
- 27.Tracey, N., Clark, J., Mander, K., McDermid, J.: An automated framework for structural test-data generation. In: ASE, p. 285. IEEE (1998)Google Scholar
- 28.Tracey, N., Clark, J.A., Mander, K.: The way forward for unifying dynamic test-case generation: the optimisation-based approach. In: Proceedings of the IFIP International Workshop on Dependable Computing and Its Applications (DCIA), York (1998)Google Scholar
- 29.Xanthakis, S., Ellis, C., Skourlas, C., Le Gall, A., Katsikas, S., Karapoulios, K.: Application of genetic algorithms to software testing. In: Proceedings of the 5th International Conference on Software Engineering and Applications, pp. 625–636 (1992)Google Scholar
- 36.Panichella, A., Kifetew, F.M., Tonella, P.: Reformulating branch coverage as a many-objective optimization problem. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10. IEEE (2015)Google Scholar
- 37.Karim, M.R., Ryan, C.: Sensitive ants are sensible ants. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 775–782. ACM (2012)Google Scholar
- 38.bibclean.c (1995). http://www.cs.bham.ac.uk/~wbl/biblio/tools/bibclean.c. Accessed 09 Nov 2018
- 40.Mitchell, M., Forrest, S., Holland, J.H.: The royal road for genetic algorithms: fitness landscapes and GA performance. In: Proceedings of the First European Conference on Artificial Life, pp. 245–254 (1992)Google Scholar