Advertisement

Single-objective Versus Multi-objectivized Optimization for Evolutionary Crash Reproduction

  • Mozhan Soltani
  • Pouria Derakhshanfar
  • Annibale Panichella
  • Xavier Devroey
  • Andy Zaidman
  • Arie van Deursen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11036)

Abstract

EvoCrash is a recent search-based approach to generate a test case that reproduces reported crashes. The search is guided by a fitness function that uses a weighted sum scalarization to combine three different heuristics: (i) code coverage, (ii) crash coverage and (iii) stack trace similarity. In this study, we propose and investigate two alternatives to the weighted sum scalarization: (i) the simple sum scalarization and (ii) the multi-objectivization, which decomposes the fitness function into several optimization objectives as an attempt to increase test case diversity. We implemented the three alternative optimizations as an extension of EvoSuite, a popular search-based unit test generator, and applied them on 33 real-world crashes. Our results indicate that for complex crashes the weighted sum reduces the test case generation time, compared to the simple sum, while for simpler crashes the effect is the opposite. Similarly, for complex crashes, multi-objectivization reduces test generation time compared to optimizing with the weighted sum; we also observe one crash that can be replicated only by multi-objectivization. Through our manual analysis, we found out that when optimizing the original weighted function gets trapped in local optima, optimization for decomposed objectives improves the search for crash reproduction. Generally, while multi-objectivization is under-explored, our results are promising and encourage further investigations of the approach.

References

  1. 1.
    Albunian, N.M.: Diversity in search-based unit test suite generation. In: Menzies, T., Petke, J. (eds.) SSBSE 2017. LNCS, vol. 10452, pp. 183–189. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-66299-2_17CrossRefGoogle Scholar
  2. 2.
    Arcuri, A., Fraser, G., Just, R.: Private API access and functional mocking in automated unit test generation. In: Proceedings of International Conference on Software Testing, Verification and Validation (ICST), pp. 126–137. IEEE (2017)Google Scholar
  3. 3.
    Bianchi, F.A., Pezzè, M., Terragni, V.: Reproducing concurrency failures from crash stacks. In: Proceedings of the Joint Meeting on Foundations of Software Engineering (FSE), pp. 705–716. ACM (2017)Google Scholar
  4. 4.
    Chen, N., Kim, S.: STAR: stack trace based automatic crash reproduction via symbolic execution. IEEE Trans. Softw. Eng. 41(2), 198–220 (2015)CrossRefGoogle Scholar
  5. 5.
    Coello Coello, C.A.: Constraint-handling techniques used with evolutionary algorithms. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO Companion), pp. 563–587. ACM (2016)Google Scholar
  6. 6.
    Črepinšek, M., Liu, S.H., Mernik, M.: Exploration and exploitation in evolutionary algorithms: a survey. ACM Comput. Surv. (CSUR) 45(3), 35 (2013)CrossRefGoogle Scholar
  7. 7.
    Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. TEVC 6(2), 182–197 (2002)Google Scholar
  8. 8.
    Feldt, R., Torkar, R., Gorschek, T., Afzal, W.: Searching for cognitively diverse tests: towards universal test diversity metrics. In: Proceedings of International Conference on Software Testing Verification and Validation Workshops (ICSTW), pp. 178–186. IEEE (2008)Google Scholar
  9. 9.
    Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Softw. Eng. 39(2), 276–291 (2013)CrossRefGoogle Scholar
  10. 10.
    Fraser, G., Arcuri, A.: A large-scale evaluation of automated unit test generation using evosuite. TOSEM 24(2), 8 (2014)CrossRefGoogle Scholar
  11. 11.
    Goldberg, D.E., Richardson, J.: Genetic algorithms with sharing for multimodal function optimization. In: Proceedings of International Conference on Genetic Algorithms and Their Application, pp. 41–49. Lawrence Erlbaum Associates Inc. (1987)Google Scholar
  12. 12.
    Harman, M., McMinn, P.: A theoretical & empirical analysis of evolutionary testing and hill climbing for structural test data generation. In: Proceedings of the 2007 International Symposium on Software Testing and Analysis, pp. 73–83. ACM (2007)Google Scholar
  13. 13.
    Jähne, M., Li, X., Branke, J.: Evolutionary algorithms and multi-objectivization for the travelling salesman problem. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 595–602. ACM (2009)Google Scholar
  14. 14.
    Jin, W., Orso, A.: Bugredux: reproducing field failures for in-house debugging. In: Proceedings of International Conference on Software Engineering (ICSE), pp. 474–484. IEEE Press (2012)Google Scholar
  15. 15.
    Just, R., Jalali, D., Ernst, M.D.: Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In: Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), pp. 437–440. ACM (2014)Google Scholar
  16. 16.
    Kifetew, F.M., Panichella, A., De Lucia, A., Oliveto, R., Tonella, P.: Orthogonal exploration of the search space in evolutionary test case generation. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA), pp. 257–267. ACM (2013)Google Scholar
  17. 17.
    Knowles, J.D., Watson, R.A., Corne, D.W.: Reducing local optima in single-objective problems by multi-objectivization. In: Zitzler, E., Thiele, L., Deb, K., Coello Coello, C.A., Corne, D. (eds.) EMO 2001. LNCS, vol. 1993, pp. 269–283. Springer, Heidelberg (2001).  https://doi.org/10.1007/3-540-44719-9_19CrossRefGoogle Scholar
  18. 18.
    McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verification Reliab. 14(2), 105–156 (2004)CrossRefGoogle Scholar
  19. 19.
    Nayrolles, M., Hamou-Lhadj, A., Tahar, S., Larsson, A.: A bug reproduction approach based on directed model checking and crash traces. J. Softw.: Evol. Process 29(3), e1789 (2017)Google Scholar
  20. 20.
    Rößler, J., Zeller, A., Fraser, G., Zamfir, C., Candea, G.: Reconstructing core dumps. In: Proceedings of International Conference on Software Testing, Verification and Validation (ICST), pp. 114–123. IEEE (2013)Google Scholar
  21. 21.
    Soltani, M., Panichella, A., van Deursen, A.: Evolutionary testing for crash reproduction. In: Proceedings of the 9th International Workshop on Search-Based Software Testing - SBST 2016, pp. 1–4 (2016)Google Scholar
  22. 22.
    Soltani, M., Panichella, A., van Deursen, A.: A guided genetic algorithm for automated crash reproduction. In: International Conference on Software Engineering (ICSE), pp. 209–220. IEEE, May 2017Google Scholar
  23. 23.
    Vargha, A., Delaney, H.D.: A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J. Educ. Behav. Stat. 25(2), 101–132 (2000)Google Scholar
  24. 24.
    Xuan, J., Xie, X., Monperrus, M.: Crash reproduction via test case mutation: let existing test cases help. In: Proceedings of the Joint Meeting on Foundations of Software Engineering (ESEC/FSE), pp. 910–913. ACM (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Delft University of TechnologyDelftThe Netherlands

Personalised recommendations