Skip to main content
SpringerLink
Log in

A pragmatic approach for hyper-parameter tuning in search-based test case generation

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Search-based test case generation, which is the application of meta-heuristic search for generating test cases, has been studied a lot in the literature, lately. Since, in theory, the performance of meta-heuristic search methods is highly dependent on their hyper-parameters, there is a need to study hyper-parameter tuning in this domain. In this paper, we propose a new metric (“Tuning Gain”), which estimates how cost-effective tuning a particular class is. We then predict “Tuning Gain” using static features of source code classes. Finally, we prioritize classes for tuning, based on the estimated “Tuning Gains” and spend the tuning budget only on the highly-ranked classes. To evaluate our approach, we exhaustively analyze 1,200 hyper-parameter configurations of a well-known search-based test generation tool (EvoSuite) for 250 classes of 19 projects from benchmarks such as SF110 and SBST2018 tool competition. We used a tuning approach called Meta-GA and compared the tuning results with and without the proposed class prioritization. The results show that for a low tuning budget, prioritizing classes outperforms the alternatives in terms of extra covered branches (10 times more than a traditional global tuning). In addition, we report the impact of different features of our approach such as search space size, tuning budgets, tuning algorithms, and the number of classes to tune, on the final results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

All Data relevant to the experiments are available in the github repository.

Code Availability

All code to reproduce the results are available in the github code repository.

Notes

  1. https://github.com/icsme2020author/Class-Prioritization-for-Tuning

References

  • Harman M, Jones BF (2001) Search-based software engineering. Inf Softw Technol 43(14):833–839

    Article  Google Scholar 

  • McMinn P (2004) Search-based software test data generation: a survey: research articles. Softw Test Verif Reliab 14(2):105–156

    Article  Google Scholar 

  • Ali S, Briand LC, Hemmati H, Kaur Panesar-Walawege R (2010) A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans Softw Eng 36(6):742–762. https://doi.org/10.1109/TSE.2009.52

    Article  Google Scholar 

  • Fraser G, Arcuri A (2011) EvoSuite: automatic test suite generation for object-oriented software. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th european conference on foundations of software engineering. ACM

  • Freisleben B, Härtfelder M (1993) Optimization of genetic algorithms by genetic algorithms. In: Artificial neural nets and genetic algorithms. Springer, Vienna, pp 392–399

  • Arcuri A, Fraser G (2013) Parameter tuning or default values? An empirical investigation in search-based software engineering. Empir Softw Eng 18 (3):594–623

    Article  Google Scholar 

  • Fraser G, Rojas JM (2018) Software testing. In: Cha S, Taylor RN, Kang KC (eds) Handbook of software engineering. Springer International Publishing

  • Agrawal A, Menzies T, Minku LL, Wagner M, Zhe Y u (2020) Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers. Empir Soft Eng 25(3):2099–2136. https://doi.org/10.1007/s10664-020-09808-9

    Article  Google Scholar 

  • Grefenstette J (1986) Optimization of control parameters for genetic algorithms. IEEE Trans Syst Man Cybern 16(1):22–128

    Article  Google Scholar 

  • Burke EK, Hyde MR, Kendall G, Ochoa G, Ozcan E, Woodward JR (2009) Exploring hyper-heuristic methodologies with genetic programming. In: Computational intelligence: collaboration, fusion and emergence. Springer, Berlin, pp 177–201

  • Crawford B, Soto R, Monfroy E, Palma W, Castro C, Paredes F (2013) Parameter tuning of a choice-function based hyperheuristic using particle swarm optimization, vol 40, pp 1690–1695

  • Craenen BGW, Eiben AE (2001) Stepwise adaption of weights with refinement and decay on constraint satisfaction problems. In: Proceedings of the 3rd annual conference on genetic and evolutionary computation. Morgan Kaufmann Publishers Inc., pp 291–298

  • Feldt R, Nordin P (2000) Using factorial experiments to evaluate the effect of genetic programming parameters. In: Proceedings of the European conference on genetic programming. Springer, pp 271–282

  • Grano G, Titov TV, Panichella S, Gall HC (2019) Branch coverage prediction in automated testing. J Softw Evol Process 31(9):e2158

    Article  Google Scholar 

  • Xu X, Zhu Z, Jiao L (2017) An adaptive fitness function based on branch hardness for search based testing. In: Proceedings of the genetic and evolutionary computation Conference. GECCO ’17. ACM, Berlin, pp 1335–1342

  • Ferrer J, Chicano F, Alba E (2013) Estimating software testing complexity. Inf Softw Technol 55(12):2125–2139

    Article  Google Scholar 

  • Zamani S, Hemmati H (2020) A cost-effective approach for hyper-parameter tuning in search-based test case generation. In: 2020 IEEE international conference on software maintenance and evolution (ICSME). https://doi.org/10.1109/ICSME46990.2020.00047, pp 418–429

  • Feurer M, Hutter F (2019) Hyperparameter optimization. In: Hutter Frank, Kotthoff Lars, Vanschoren Joaquin (eds) Automated machine learning: methods, systems, challenges. https://doi.org/10.1007/978-3-030-05318-51. Springer International Publishing, Cham, pp 3–33

  • Zamani S, Hemmati H (2019) Revisiting hyper-parameter tuning for search-based test data generation. In: Nejati Shiva, Gay Gregory (eds) Search-based software engineering. Springer International Publishing, Cham, pp 137–152

  • Harman M (2007) The current state and future of search based software engineering. In: Future of software engineering (FOSE ’07), pp 342–357

  • Sakti A, Pesant G, Guéhéneuc Y-G (2014) Instance generator and problem representation to improve object oriented code coverage, vol 41, pp 294–313

  • Oster N, Saglietti F (2006) Automatic test data generation by multi-objective optimisation conference=Proceedings of the 25th international conference on computer, safety, reliability, and security. SAFECOMP’06. https://doi.org/10.1007/1187556732. Springer, Gdansk, pp 426–438

  • Eiben ÁE, Hinterding R, Michalewicz Z (1999) Parameter control in evolutionary algorithms, vol 3, pp 124–141

  • Shu R, Xia T, Chen J, Williams L, Menzies T (2021) How to better distinguish security bug reports (using dual hyperparameter optimization). Empir Softw Eng 26. https://doi.org/10.1007/s10664-020-09906-8

  • Bergstra J, Bardenet R, Bengio Y, Kégl B (2011). In: Proceedings of the 24th international conference on neural information processing systems. NIPS’11. Curran Associates Inc., Granada, pp 2546–2554

  • Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2016) Automated parameter optimization of classification techniques for defect prediction models. In: Proceedings of the 38th international conference on software engineering, pp 321–332

  • Bergstra J, Bengio Y (2012) Random search for hyper parameter optimization. J Mach Learn Res 13(null):281–305

    MathSciNet  MATH  Google Scholar 

  • Goldberg DE (2006) Genetic algorithms. Pearson Education India, Bengaluru

    Google Scholar 

  • Kirkpatrick S, Daniel Gelatt C, Vecchi MP (1983) Optimization by simulated annealing, vol 220, pp 671–680

  • Pelikan M, Goldberg DE, Cantú-Paz E et al (1999) BOA: The Bayesian optimization algorithm. In: Proceedings of the genetic and evolutionary computation conference GECCO-99. volume= 1,. Citeseer, pp 525–532

  • van Stein B, Wang H, Bäck T. (2019) Automatic configuration of deep neural networks with parallel efficient global optimization. In: 2019 international joint conference on neural networks (IJCNN). IEEE, pp 1–7

  • Storn R, Price K (1997) Differential evolution – a simple and efficient heuristic for global optimization over continuous spaces. https://doi.org/10.1023/A:1008202821328, vol 11, pp 341–359

  • Agrawal A, Menzies T (2018) Is “better data” better than “better data miners”? on the benefits of tuning SMOTE for defect prediction. In: Proceedings of the 40th international conference on software engineering. ICSE ’18. Association for Computing Machinery, Gothenburg. https://doi.org/10.1145/3180155.3180197, pp 1050–1061

  • Agrawal A, Fu W, Menzies T (2018) What is wrong with topic modeling? And how to fix it using search-based software engineering. https://www.sciencedirect.com/science/article/pii/S0950584917300861, vol 98, pp 74–88, DOI https://doi.org/10.1016/j.infsof.2018.02.005

  • Fu W, Nair V, Menzies T (2016) Why is differential evolution better than grid search for tuning defect predictors?. In: CoRR. http://dblp.uni-trier.de/db/journals/corr/corr1609.html#FuNM16

  • Wei F u, Menzies T, Shen X (2016) Tuning for Software Analytics, vol 76, pp 135–146, DOI https://doi.org/10.1016/j.infsof.2016.04.017

  • Chen D, Fu W, Krishna R, Menzies T (2018) Applications of psychological science for actionable analytics. In: Proceedings of the 2018 26th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. ESEC/FSE 2018. Association for Computing Machinery, Lake Buena Vista. https://doi.org/10.1145/3236024.3236050, pp 456–467

  • Birattari M (2009) Tuning metaheuristics: a machine learning perspective, 1st edn. Springer Publishing Company, Incorporated, Berlin. 2nd printing

    Book  Google Scholar 

  • Myers RH, Montgomery DC (1995) Response surface methodology: process and product in optimization using designed experiments, 1st edn. Wiley, Hoboken

    MATH  Google Scholar 

  • Sayyad AS, Goseva-Popstojanova K, Menzies T, Ammar H (2013) On parameter tuning in search based software engineering: a replicated empirical study. In: Proceedings of workshop on replication in empirical software engineering. IEEE Computer Society

  • Kotelyanskii A, Kapfhammer GM (2014) Parameter tuning for search-based test-data generation revisited: support for previous results. In: Proceedings of the 2014 14th international conference on quality software. IEEE Computer Society, pp 79–84

  • Song L, Minku LL, Yao X (2013) The impact of parameter tuning on software effort estimation using learning machines. In: Proceedings of the 9th international conference on predictive models in software engineering. ACM, pp 9:1–9:10

  • Xia T, Krishna R, Chen J, Mathew G, Shen X, Menzies T (2020) Sequential model optimization for software effort estimation. IEEE Trans Softw Eng :1–1, DOI https://doi.org/10.1109/TSE.2020.3047072, (to appear in print)

  • Fu W, Menzies T, Shen X (2016) Tuning for software analytics: Is it really necessary?. Inf Softw Technol 76:135–146

    Article  Google Scholar 

  • Wang T, Harman M, Jia Y, Krinke J (2013) Searching for better configurations: a rigorous approach to clone evaluation. In: Proceedings of the 2013 9th joint meeting on foundations of software engineering. ACM, pp 455–465

  • Panichella A, Dit B, Oliveto R, Di Penta M, Poshynanyk D, De Lucia A (2013) How to effectively use topic models for software engineering tasks? an approach based on genetic algorithms. In: 2013 35th international conference on software engineering (ICSE). IEEE, pp 522–531

  • Treude C, Wagner M (2019) Predicting good configurations for GitHub and stack overflow topic models. In: 2019 IEEE/ACM 16th international conference on mining software repositories (MSR). https://doi.org/10.1109/MSR.2019.00022, pp 84–95

  • Agrawal A, Fu W, Chen D, Shen X, Menzies T (2019) How to ”DODGE” complex software analytics. IEEE Trans Softw Eng :1–1. https://doi.org/10.1109/TSE.2019.2945020

  • Zhang Y, Harman M, Ochoa G, Ruhe G, Brinkkemper S (2018) An empirical study of meta and hyper-heuristic search for multi-objective release planning. ACM Trans Softw Eng Methodol 27(1), DOI https://doi.org/10.1145/3196831, (to appear in print)

  • Smit SK, Eiben AE (2009) Comparing parameter tuning methods for evolutionary algorithms

  • Clarkware (2020) Jdepend. https://github.com/clarkware/jdepend

  • Aniche M (2015) Java code metrics calculator (CK). https://github.com/mauricioaniche/ck/

  • McCabe TJ (1976) A complexity measure, vol 2, pp 308–320

  • Eder L (2021) KeyWord counter. https://github.com/lukaseder/silly-metrics

  • Metwally AA (2021) Halstead complexitymeasures. https://github.com/aametwally/Halstead-Complexity-Measureshttps://github.com/aametwally/Halstead-Complexity-Measures

  • Fraser G, Arcuri A (2014) A large-scale evaluation of automated unit test generation using EvoSuite. ACM Trans Softw Eng Methodol 24(2):8:1–8:42

    Article  Google Scholar 

  • Panichella A, Kifetew F, Tonella P (2017) Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Trans Softw Eng 44(2):122–158

    Article  Google Scholar 

  • Rueda Molina U, Kifetew F, Panichella A (2018) Java unit testing tool competition - sixth round. In: 2018 IEEE/ACM 11th international workshop on search-based software testing (SBST), pp 22–29

  • Fraser G, Arcuri A (2013) Whole test suite generation, vol 39, pp 276–291

  • Tange O (2018) GNU Parallel 2018. Lulu.com

  • Järvelin K, Kekäläinen J (2002) Cumulated gain-based evaluation of IR techniques. ACM Trans Inf Syst 20(4):422–446. https://doi.org/10.1145/582415.582418

    Article  Google Scholar 

  • Guyon I, Elisseeff A (2006) An introduction to feature extraction. In: Guyon by Isabelle, Nikravesh Masoud, Gunn Steve, Zadeh LA (eds) Feature extraction: foundations and applications. Springer, Berlin, pp 1–25

  • Paterson D, Turner J, White T, Fraser G (2015) Parameter control in search-based generation of unit test suites. In: Barros M, Labiche Y (eds) Search-based software engineering. Springer International Publishing, Cham, pp 141–156. Title Suppressed Due to Excessive Length 43

  • Price K, Storn RM, Lampinen JA (2006) Differential evolution: a practical approach to global optimization. Springer Science and Business Media, Berlin

    MATH  Google Scholar 

  • Vassilev VK, Fogarty TC, Miller JF (2003) Smoothness, ruggedness and neutrality of fitness landscapes: from theory to application. In: Ghosh A, Tsutsui S (eds) Advances in evolutionary computing: theory and applications. Springer, Berlin, pp 3–44. https://doi.org/10.1007/978-3-642-18965-41

  • Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: Machine Learning in Python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  • Malan KM, Engelbrecht AP (2013) A survey of techniques for characterising fitness landscapes and some possible ways forward. Inf Sci 241:148–163. https://doi.org/10.1016/j.ins.2013.04.015

    Article  Google Scholar 

Download references

Funding

This work is partially supported by Compute Canada, the Natural Sciences and Engineering Research Council of 1049 Canada [RGPIN/04552-2020] and Mitacs Accelerate Program [Application Ref. IT16572].

Author information

Authors and Affiliations

  1. Department of Electrical and Software Engineering, Schulich School of Engineering, University of Calgary, Calgary, Canada

    Shayan Zamani & Hadi Hemmati

Authors
  1. Shayan Zamani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hadi Hemmati
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shayan Zamani.

Ethics declarations

Conflict of Interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Communicated by: Tim Menzies

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zamani, S., Hemmati, H. A pragmatic approach for hyper-parameter tuning in search-based test case generation. Empir Software Eng 26, 126 (2021). https://doi.org/10.1007/s10664-021-10024-2

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-10024-2

Keywords

Search

Navigation