Skip to main content
SpringerLink
Log in

An extensive evaluation of search-based software testing: a review

  • Methodologies and Application
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

In recent years, search-based software testing (SBST) is the active research topic in software testing. SBST is the process of generating test cases that use metaheuristics for optimization of a task in the framework of software testing to solve difficult NP-hard problems. The best fitness results must be found with the heuristic search among many possibilities for a more cost-effective testing process and automate the process of generating test cases. Although search-based test data generation is a field of interest, some challenges remain unknown. The main objective of this survey is to find the main topics and trends in this emerging field of search-based software testing by examining the methods and the literature of software testing. A review of earlier studies of search-based software testing from the year 1996 to 2016 is discussed with the application of metaheuristics for the optimization of software testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Afzal W, Torkar R, Feldt R (2009) A systematic review of search-based testing for non-functional system properties. Inf Softw Technol 51:957–976

    Article  Google Scholar 

  • Afzal W, Alone S, Glocksien K, Torkar R (2016) Software test process improvement approaches. J Syst Softw 111:1–33

    Article  Google Scholar 

  • Alander J, Mantere T, Turunen P (1998) Genetic algorithm based software testing. Springer, Vienna, pp 325–328

    Google Scholar 

  • Alba E, Chicano FJ (2006) Software testing with evolutionary strategies. Springer, Vienna, pp 50–65

    Google Scholar 

  • Alégroth E, Feldt R, Kolström P (2016) Maintenance of automated test suites in industry: an empirical study on Visual GUI Testing. Inf Softw Technol 73:66–80

    Article  Google Scholar 

  • Ali S, Briand LC, Hemmati H, Rajwinder K, Panesar-Walawege RK (2010) A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans Softw Eng 36(6):742–762

    Article  Google Scholar 

  • Arcuri A (2010) It does matter how you normalise the branch distance in search based software testing. IEEE, pp 205–214

  • Bauersfeld S, Wappler S, Wegener J (2011) A metaheuristic approach to test sequence generation for applications with a GUI. Springer, vol 69(56), pp 173–187

  • Bertolino A (2007) Software testing research. In: Achievements, challenges, dreams. IEEE, pp 85–103

  • Daka E, Campos J, Fraser G, Dorn J, Weimer W (2015) Modeling readability to improve unit tests. ACM, pp 107–118

  • Díaz E, Tuya J, Blanco R (2003) Automated software testing using a metaheuristic technique based on Tabu search. IEEE, pp 310–313

  • Dobuneh N, Raeisi M, Jawawi DN, Ghazali M, Malakooti MV (2014) Development test case prioritization technique in regression testing based on hybrid criteria. IEEE, pp 301–305

  • Eickelmann NS, Richardson DJ (1996) An evaluation of software test environment architectures. IEEE, pp 353–364

  • El-Serafy A, El-Sayed G, Salama C, Wahba(2015) An enhanced genetic algorithm for MC/DC test data generation. IEEE, pp 1–8

  • Ferguson R, Korel B (1996) The chaining approach for software test data generation. ACM Trans Softw Eng Methodol 5(1):63–86

    Article  Google Scholar 

  • Fin A, Fummi F, Pravadelli G, Amleto (2001) A multi-language environment for functional test generation. IEEE, pp 821–829

  • Fraser G, Zeller A (2012) Mutation-driven generation of unit tests and oracles. IEEE Trans Softw Eng 38(2):278–292

    Article  Google Scholar 

  • Fraser G, Arcuri A (2012) Sound empirical evidence in software testing. IEEE, pp 178–188

  • Gallagher MJ, Narasimhan VL (1997) Adtest: a test data generation suite for ada software systems. IEEE Trans Softw Eng 23(8):473–484

    Article  Google Scholar 

  • Gotlieb A, Botella B, Rueher M (1998) Automatic test data generation using constraint solving techniques. ACM SIGSOFT 23(2)

  • Gupta N, Mathur AP, Soffa ML (1998) Automated test data generation using an iterative relaxation method. ACM SIGSOFT Softw Eng Notes 23(6):231–244

    Article  Google Scholar 

  • Harman M (2007) Search based software engineering for program comprehension. Program Comprehension. IEEE

  • Harman M (2008) Open problems in testability transformation. IEEE, pp 196–209

  • Harman M, Hassoun Y, Lakhotia K, McMinn P, Wegener J (2007) The impact of input domain reduction on search-based test data generation. ACM SIGSOFT, pp 155–164

  • Harman M, Jia Y, Zhang Y (2015) Achievements, open problems and challenges for search based software testing. In: Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference. IEEE, pp 1–12

  • Harman M, McMinn P (2010) A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE, vol 36(2), pp 226–247

  • Harman M, McMinn P, De Souza JT, Yoo S (2012) Search based software engineering: techniques, taxonomy, tutorial. Springer, vol 7007, pp 1–59

  • Kapfhammer GM, McMinn P, Chris J (2013) Wright. Search-based testing of relational schema integrity constraints across multiple database management systems. IEEE. pp 31–40

  • Korel B (1996) Automated test data generation for programs with procedures. ACM SIGSOFT, vol 21(3), pp 209–215

  • Korel B, Harman M, Chung S, Apirukvorapinit P, Gupta R, Zhang Q (2005) Data dependence based testability transformation in automated test generation. IEEE, pp 245–254

  • Kos T, Mernik M, Kosar T (2016) Test automation of a measurement system using a domain-specific modelling language. J Syst Softw 111:74–88

    Article  Google Scholar 

  • Kotelyanskii A, Kapfhammer GM (2014) Parameter tuning for search-based test-data generation revisited: Support for previous results. IEEE, pp 79–84

  • Kuhn DR, Wallace DR, Gallo AM Jr (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421

    Article  Google Scholar 

  • Lakhotia K McMinn P, Harman M (2009) Automated test data generation for coverage: Haven’t we solved this problem yet? IEEE, pp 95–104

  • Lefticaru R, Ipate F (2008) Functional search-based testing from state machines. IEEE, pp 525–528

  • Mahali P, Acharya AA, Mohapatra DP (2016) Test case prioritization using association rule mining and business criticality test value. Springer, vol 2, pp 335–345

  • Malhotra R, Khari M (2013) Heuristic search-based approach for automated test data generation: a survey. Int J Bio-Inspired Comput 5(1):1–18

    Article  Google Scholar 

  • Marín B, Gallardo C, Quiroga D, Giachetti G, Serral E (2016) Testing of model-driven development applications. Springer, New York, pp 1–29

    Google Scholar 

  • Martins E, Sabião SB, Ambrosio AM (1999) ConData: a tool for automating specification-based test case generation for communication systems. Softw Qual J 8(4):303–320

    Article  Google Scholar 

  • McMinn P, Binkley D, Harman M (2009) Empirical evaluation of a nesting testability transformation for evolutionary testing. ACM Trans Softw Eng Methodol 18(3):1–27

    Article  Google Scholar 

  • McMinn P, Harman M, Lakhotia K, Hassoun Y, Wegener J (2012) Input domain reduction through irrelevant variable removal and its effect on local, global, and hybrid search-based structural test data generation. IEEE Trans Softw Eng 38(2):453–477

    Article  Google Scholar 

  • McMinn P, Harman M, Binkley D, Tonella P (2006) The species per path approach to search based test data generation. ACM, pp 13–24

  • McMinn P, Shahbaz M, Stevenson M (2012) Search-based test input generation for string data types using the results of web queries. IEEE, pp 141–150

  • Michael CC, McGraw G (1998) Automated software test data generation for complex programs. IEEE, pp 136–146

  • Michael CC, McGraw GE, Schatz MA, Walton CC (1997) Genetic algorithms for dynamic test data generation. IEEE, pp 307–308

  • Michael CC, McGraw G, Schatz MA (2001) Generating software test data by evolution. Ser Softw Eng Knowl Eng 27(12):1085–1110

    Google Scholar 

  • Miller W, Spooner D (1976) Automatic generation of floating point test data. IEEE Trans Softw Eng 2(3):223–226

    Article  MathSciNet  Google Scholar 

  • Moadab S, Rashidi H (2016) Automatic path-oriented test data generation by boundary hyper cuboids. J King Saud Univ Comput Inf Sci 28(1):82–97

    Article  Google Scholar 

  • Nguyen Tony, Nassif N (2016) Optimization of HVAC systems using genetic algorithm. Springer, New York, pp 203–209

    Google Scholar 

  • Nguyen TB, Delaunay M, Robach C (2005) Testability analysis of data-flow software. Electron Notes Theor Comput Sci 116:213–225

    Article  Google Scholar 

  • Patrick M (2016) Metaheuristic optimisation and mutation-driven test data generation. Springer, pp 89–115

  • Prabu M, Narasimhan D, Raghuram S (2016) An effective tool for optimizing the number of test paths in data flow testing for anomaly detection. Springer, pp 505–518

  • Priyanka, Inderveer, Rana A (2016) Cloud-based automatic test data generation framework. Elsevier, pp 1–33

  • Rauf A, Anwar S, Jaffer MA (2010) Automated GUI test coverage analysis using GA. IEEE, pp 1057–1062

  • Rogstad E, Briand L (2016) Cost-effective strategies for the regression testing of database applications: case study and lessons learned. J Syst Softw 113:257–274

    Article  Google Scholar 

  • Salman YD, Hashim NL (2016) Automatic test case generation from UML state chart diagram: a survey. Springer, vol 362, pp 123–134

  • Shahbaz M, McMinn P, Stevenson M (2015) Automatic generation of valid and invalid test data for string validation routines using web searches and regular expressions. Sci Comput Program 97:405–425

    Article  Google Scholar 

  • Shah S, Sudarshan S, Kajbaje S, Patidar S, Gupta BP, Vira D (2011) Generating test data for killing SQL mutants: a constraint-based approach. IEEE, pp 1175–1186

  • Shen X, Wang Q, Wang P, Zhou B (2009) Automatic generation of test case based on GATS algorithm. IEEE, pp 496–500

  • Sofokleous AA, Andreas AS (2007) Batch-optimistic test-cases generation using genetic algorithms. IEEE, vol 1, pp 157–164

  • Sofokleous AA, Andreou AS (2008) Automatic, evolutionary test data generation for dynamic software testing. J Syst Softw 81(11):1883–1898

    Article  Google Scholar 

  • Utting M, Legeard B, Bouquet F, Fourneret E, Peureux F, Vernotte A (2016) Recent advances in model-based testing. Adv Comput 101:53–120

    Article  Google Scholar 

Download references

Acknowledgements

No funding is provided for the preparation of manuscript.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, National Institute of Technology, Patna, India

    Manju Khari & Prabhat Kumar

Authors
  1. Manju Khari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Prabhat Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manju Khari.

Ethics declarations

Conflict of interest

Authors Manju Khari and Dr. Prabhat Kumar declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Communicated by V. Loia.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khari, M., Kumar, P. An extensive evaluation of search-based software testing: a review. Soft Comput 23, 1933–1946 (2019). https://doi.org/10.1007/s00500-017-2906-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-017-2906-y

Keywords

Search

Navigation