Using Search-Based Test Generation to Discover Real Faults in Guava

  • Hussein Almulla
  • Alireza Salahirad
  • Gregory Gay
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10452)


Testing costs can be reduced through automated unit test generation. An important benchmark for such tools is their ability to detect real faults. Fault databases, such as Defects4J, assist in this task. The Guava project—a collection of Java libraries from Google—offers an opportunity to expand such databases with additional complex faults. We have identified 11 faults in the Guava project, added them to Defects4J, and assessed the ability of the EvoSuite framework to detect these faults. Ultimately, EvoSuite was able to detect three faults. Analysis of the remaining faults offers lessons in how to improve generation tools. We offer these faults to the community to assist future benchmarking efforts.


Search-based test generation Automated test generation Software faults 


  1. 1.
    Feldt, R., Poulding, S.: Finding test data with specific properties via metaheuristic search. In: 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), pp. 350–359, November 2013Google Scholar
  2. 2.
    Fraser, G., Arcuri, A.: Automated test generation for java generics. In: Winkler, D., Biffl, S., Bergsmann, J. (eds.) SWQD 2014. LNBIP, vol. 166, pp. 185–198. Springer, Cham (2014). doi: 10.1007/978-3-319-03602-1_12 CrossRefGoogle Scholar
  3. 3.
    Fraser, G., Staats, M., McMinn, P., Arcuri, A., Padberg, F.: Does automated white-box test generation really help software testers? In: Proceedings of the 2013 International Symposium on Software Testing and Analysis, ISSTA, pp. 291–301. ACM, New York (2013).
  4. 4.
    Gay, G.: The fitness function for the job: search-based generation of test suites that detect real faults. In: Proceedings of the International Conference on Software Testing, ICST 2017. IEEE (2017)Google Scholar
  5. 5.
    Gay, G.: Generating effective test suites by combining coverage criteria. In: Menzies, T., Petke, J. (eds.) SSBSE 2017. LNCS, vol. 10452, pp. 65–82. Springer, Cham (2017)Google Scholar
  6. 6.
    Just, R., Jalali, D., Ernst, M.D.: defects4J: a database of existing faults to enable controlled testing studies for Java programs. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pp. 437–440. ACM, New York (2014).
  7. 7.
    Shamshiri, S., Just, R., Rojas, J.M., Fraser, G., McMinn, P., Arcuri, A.: Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges. In: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), ASE 2015. ACM, New York (2015)Google Scholar
  8. 8.
    Weiss, T.: We analyzed 30,000 GitHub projects - here are the top. 100 libraries in Java, JS and Ruby (2013).

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Hussein Almulla
    • 1
  • Alireza Salahirad
    • 1
  • Gregory Gay
    • 1
  1. 1.University of South CarolinaColumbiaUSA

Personalised recommendations