Analysing the fitness landscape of search-based software testing problems
Search-based software testing automatically derives test inputs for a software system with the goal of improving various criteria, such as branch coverage. In many cases, evolutionary algorithms are implemented to find near-optimal test suites for software systems. The result of the search is usually received without any indication of how successful the search has been. Fitness landscape characterisation can help understand the search process and its probability of success. In this study, we recorded the information content, negative slope coefficient and the number of improvements during the progress of a genetic algorithm within the EvoSuite framework. Correlating the metrics with the branch and method coverages and the fitness function values reveals that the problem formulation used in EvoSuite could be improved by revising the objective function. It also demonstrates that given the current formulation, the use of crossover has no benefits for the search as the most problematic landscape features are not the number of local optima but the presence of many plateaus.
KeywordsTest data generation Genetic algorithms Fitness landscape characterisation
We would like to thank Gordon Fraser and his team who developed EvoSuite for making the source code available. We wish to acknowledge Monash University for the use of their Nimrod software in this work. The Nimrod project has been funded by the Australian Research Council and a number of Australian Government agencies, and was initially developed by the Distributed Systems Technology. This research was supported under Australian Research Council’s Discovery Projects funding scheme (Project Number DE140100017).
Compliance with ethical standards
Conflicts of interest
The authors declare that they have no conflict of interest.
This research does not involve human participants and animals.
- Arcuri, A., Fraser, G.: On parameter tuning in search based software engineering. In: SSBSE, pp. 33–47 (2011)Google Scholar
- Gheorghita, M., Moser, I., Aleti, A.: Characterising fitness landscapes using predictive local search. In: Proceedings of the 15th annual conference companion on genetic and evolutionary computation, pp. 67–68. ACM (2013a)Google Scholar
- Gheorghita, M., Moser, I., Aleti, A.: Designing and characterising fitness landscapes with various operators. In: Evolutionary computation (CEC), 2013 IEEE congress on, pp. 2766–2772 (2013b)Google Scholar
- Godefroid, P., Klarlund, N., Sen, K.: Dart: Directed automated random testing. In: Proceedings of the 2005 ACM SIGPLAN conference on programming language design and implementation, PLDI ’05, pp. 213–223. ACM (2005)Google Scholar
- Hierons, R.M., Bogdanov, K., Bogdanov, J.P., Cleaveland, R., Derrick, J., Dick, J., Gheorghe, M., Harman, M., Kapoor, K., Krause, P.J., Lüttgen, G., Simons, A.J.H., Vilkomir, S.A., Woodward, M.R., Zedan, H.: Using formal specifications to support testing. ACM Comput. Surv. 41(2), 9 (2009)CrossRefGoogle Scholar
- Kalboussi, S., Bechikh, S., Kessentini, M., Said, L.B.: Preference-based many-objective evolutionary testing generates harder test cases for autonomous agents. In: Ruhe, G., Zhang, Y., (eds.) Search based software engineering—5th international symposium, SSBSE 2013. Lecture notes in computer science, vol. 8084, pp. 245–250. Springer, New YorkGoogle Scholar
- Kinnear, K.E.: Fitness landscapes and difficulty in genetic programming. In: IEEE conference on evolutionary computation, vol. 1, pp. 142–47. IEEE (1994)Google Scholar
- Pitzer, E., Affenzeller, M.: A comprehensive survey on fitness landscape analysis. In: Fodor, J., Klempous, J.C.R., Araujo, C.P.S. (eds.) Recent Advances in Intelligent Engineering Systems of Studies in Computational Intelligence, pp. 161–191. Springer, New York (2012)Google Scholar
- Roper, M.: Computer aided software testing using genetic algorithms. 10th international quality week (1997)Google Scholar
- Sen, K., Marinov, D., and Agha, G.: Cute: a concolic unit testing engine for c. In: Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on foundations of software engineering, ESEC/FSE-13, pp. 263–272, ACM, New York (2005)Google Scholar
- Shannon, C., Weaver, W.: The Mathematical Theory of Communication. Number pt. 11 in Illini Books. University of Illinois Press, Champaign (1963)Google Scholar
- Tillmann, N. De Halleux, J.: Pex: white box test generation for net. In: Proceedings of the 2nd international conference on tests and proofs, TAP’08, pp. 134–153. Springer, New York (2008)Google Scholar
- Vanneschi, L., Tomassini, M., Collard, P., Vérel, S.: Negative slope coefficient: a measure to characterize genetic programming fitness landscapes. Genetic Programming. Lecture Notes in Computer Science, vol. 3905, pp. 178–189. Springer, Berlin (2006)Google Scholar
- Vivanti, M., Mis, A., Gorla, A., Fraser, G.: Search-based data-flow test generation. In: IEEE 24th international symposium on software reliability engineering, ISSRE 2013, pp. 370–379. IEEE Computer Society (2013)Google Scholar
- Watkins, A.L.: The automatic generation of test data using genetic algorithms. In: Proceedings of the 4th software quality conference, vol. 2, pp. 300–309 (1995)Google Scholar
- Whitley, D.: The GENITOR algorithm and selection pressure: why rank-based allocation. In: Schaffer, J.D. (ed) Proceedings of the third international conference on genetic algorithms, pp. 116–121, San Mateo, Morgan Kaufmann (1989)Google Scholar
- Williams, N., Marre, B., Mouy, P., and Roger, M.: Pathcrawler: Automatic generation of path tests by combining static and dynamic analysis. In: Cin, M. D., Kaâniche, M., Pataricza, A., (eds) Proceedings of dependable computing—EDCC-5, 5th European dependable computing conference, budapest, Hungary, April 20–22, 2005, Lecture notes in computer science, vol. 3463, pp. 281–292. Springer, New York (2005)Google Scholar
- Xanthakis, S., Ellis, C., Skourlas, C., Le Gall, A., Katsikas, S., and Karapoulios, K.: Application of genetic algorithms to software testing. In: Proceedings of the 5th international conference on software engineering and its applications, pp. 625–636 (1992)Google Scholar
- Xin, B., Chen, J., and Pan, F.: Problem difficulty analysis for particle swarm optimization: deception and modality. In: Proceedings of the first ACM/SIGEVO summit on genetic and evolutionary computation, pp. 623–630. ACM (2009)Google Scholar