Software testing is one of the most labor-intensive and crucial phases of the software development life cycle. It accounts for about 50–60% of the total cost of a project and requires a substantial amount of effort to ensure that the quality of the software is not compromised. Hence, optimizing the process of software testing has been of paramount importance in the IT industry. In recent times, there has been a significant shift toward software test automation to ensure the quality of the software while minimizing the cost and effort. This study proposes a statistical model analyzing the various factors that affect the testing cost, quality and effort by studying the behavioral pattern of professionals employed in the field of testing. The paper studies the factors, project duration, Software Complexity (SC), testing tools and automated test suite generation, while contrasting it with the overall software test automation, with the aim of finding a measure of how much automation affects the cost, quality and effort in practice. The results indicated that automation accounted for a variation of about 25% in each of the factors: cost, quality and effort much more than factors such as project duration, SC and testing tool . Further, the analysis indicated that the effect of SC on cost and effort was extremely low. This suggests that automation improved the quality while reducing the cost and effort irrespective of the complexity of the software being developed.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Cohen, C.F.; Birkin, S.J.; Garfield, M.J.; Webb, H.W.: Managing conflict in software testing. Commun. ACM 47(1), 76–81 (2004)
Planning, S.: The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology (2002)
Poon, P.L.; Tse, T.H.; Tang, S.F.; Kuo, F.C.: Contributions of tester experience and a checklist guideline to the identification of categories and choices for software testing. Softw. Qual. J. 19(1), 141–163 (2011)
Boehm, B.W.; Brown, J.R.; Lipow, M.: Quantitative evaluation of software quality. In: Proceedings of the 2nd International Conference on Software Engineering, pp. 592–605. IEEE Computer Society Press (1976)
Horgan, J.R.; London, S.; Lyu, M.R.: Achieving software quality with testing coverage measures. Computer 27(9), 60–69 (1994)
Sthamer, H.H.: The automatic generation of software test data using genetic algorithms. Doctoral dissertation, University of Glamorgan (1995)
Ghiduk, A.S.; Girgis, M.R.: Using genetic algorithms and dominance concepts for generating reduced test data. Informatica 34(3), 377–385 (2010)
Prasanna, M.; Sivanandam, S.; Venkatesan, R.; Sundarrajan, R.: A survey on automatic test case generation. Acad. Open Internet J. 15(6) (2005)
Tran, H.; Chechik, M.: Test generation using model checking. In: European Conference on Software Maintenance and Reengineering, CSMR2001 (2001). http://www.cs.toronto.edu/~chechik/courses00/csc2108/projects/4.pdf
Utting, M.; Pretschner, A.; Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verifi. Reliab. 22(5), 297–312 (2012)
McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verifi. Reliab. 14(2), 105–156 (2004)
Harrold, M.J.: Testing: a roadmap. In: Proceedings of the Conference on the Future of Software Engineering, pp. 61–72. ACM (2000)
Bertolino, A.: Software testing research: achievements, challenges, dreams. In: 2007 Future of Software Engineering, pp. 85–103. IEEE Computer Society (2007)
Woodward, M.R.; Hedley, D.; Hennell, M.A.: Experience with path analysis and testing of programs. IEEE Trans. Softw. Eng. 3, 278–286 (1980)
Clarke, L.A.: A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng. 3, 215–222 (1976)
McCabe, T.J.: A complexity measure. IEEE Trans. Softw. Eng. 4, 308–320 (1976)
Prather, R.E.; Myers, J.P.: The path prefix software testing strategy. IEEE Trans. Softw. Eng. 7, 761–766 (1987)
Korel, B.: Automated software test data generation. IEEE Trans. Softw. Eng. 16(8), 870–879 (1990)
Frhlich, P.; Link, J.: Automated test case optimization from dynamic models. In: ECOOP 2000 Object-Oriented Programming, pp. 472–491. Springer, Berlin (2000)
Gotlieb, A.; Petit, M.: Path-oriented random testing. In: Proceedings of the 1st International Workshop on Random Testing, pp. 28–35. ACM (2006)
Afzal, W.; Torkar, R.; Feldt, R.: A systematic review of search-based testing for non-functional system properties. Inf. Softw. Technol. 51(6), 957–976 (2009)
Harman, M.; Mansouri, S.A.; Zhang, Y.: Search based software engineering: a comprehensive analysis and review of trends techniques and applications. Department of Computer Science, King’s College London, Tech. Rep. TR-09-03, 23 (2009)
Khari, M.; Kumar, P.: An effective meta-heuristic cuckoo search algorithm for test suite optimization. Informatica 41(3), 363–377 (2017)
Khari, M.; Kumar, P.: Evolutionary computation-based techniques over multiple data sets: an empirical assessment. Arab. J. Sci. Eng. 43(8), 3875–3885 (2018)
Khari, M.; Kumar, P.; Burgos, D.; Crespo, R.G.: Optimized test suites for automated testing using different optimization techniques. Soft Comput. 22(24), 8341–8352 (2018)
Kumar, P.; Khari, M.: An investigating approach for optimization of software test suite. Recent Adv. Commun. Netw. Technol. (Former. Recent Pat. Telecommun.) 6(1), 41–54 (2017)
Sofokleous, A.A.; Andreou, A.S.: Automatic, evolutionary test data generation for dynamic software testing. J. Syst. Softw. 81(11), 1883–1898 (2008)
Alba, E.; Chicano, F.: Observations in using parallel and sequential evolutionary algorithms for automatic software testing. Comput. Oper. Res. 35(10), 3161–3183 (2008)
Díaz, E.; Tuya, J.; Blanco, R.; Dolado, J.J.: A tabu search algorithm for structural software testing. Comput. Oper. Res. 35(10), 3052–3072 (2008)
Harman, M.: The current state and future of search based software engineering. In: 2007 Future of Software Engineering, pp. 342–357. IEEE Computer Society (2007)
Li, H.; Lam, C.P.: Software test data generation using ant colony optimization. In: International Conference on Computational Intelligence, pp. 1–4 (2004)
Tahbildar, H.; Kalita, B.: Automated test data generation based on individual constraints and boundary value. Int. J. Comput. Sci. Issues (IJCSI) 7(5), 350 (2010)
Briand, L.; Labiche, Y.: A UML-based approach to system testing. Softw. Syst. Model. 1(1), 10–42 (2002)
Patil, S.S.: Functional test case generation based on model driven testing using FSM and UML activity diagram. Int. J. Adv. Res. Comput. Sci. 8(5), 1527–1530 (2017)
Ferrari, F.C.; Nakagawa, E.Y.; Rashid, A.; Maldonado, J.C.: Automating the mutation testing of aspect-oriented Java programs. In: Proceedings of the 5th Workshop on Automation of Software Test, pp. 51–58. ACM (2010)
Guderlei, R.; Just, R.; Schneckenburger, C.; Schweiggert, F.: Benchmarking testing strategies with tools from mutation analysis. In: 2008 IEEE International Conference on Software Testing Verification and Validation Workshop, pp. 360–364. IEEE (2008)
Qu, X.; Cohen, M.B.; Rothermel, G.: Configuration-aware regression testing: an empirical study of sampling and prioritization. In: Proceedings of the 2008 International Symposium on Software Testing and Analysis, pp. 75–86. ACM (2008)
Saha, G.K.: Understanding software testing concepts. Ubiquity 2008(February), 2 (2008)
Ali, S.; Briand, L.C.; Hemmati, H.; Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36(6), 742–762 (2009)
Devaraj, D.; Roselyn, J.P.: Genetic algorithm based reactive power dispatch for voltage stability improvement. Int. J. Electr. Power Energy Syst. 32(10), 1151–1156 (2010)
Harman, M.; McMinn, P.: A theoretical and empirical study of search-based testing: local, global, and hybrid search. IEEE Trans. Softw. Eng. 36(2), 226–247 (2009)
Hermadi, I.; Lokan, C.; Sarker, R.: Genetic algorithm based path testing: challenges and key parameters. In: 2010 Second World Congress on Software Engineering, vol. 2, pp. 241–244 . IEEE (2010)
Michaelis, M.: Boon and bane of GUI test automation. Test. Exp. 17, 25–28 (2008)
Kaur, H.; Gupta, G.: Comparative study of automated testing tools: selenium, quick test professional and testcomplete. Int. J. Eng. Res. Appl. 3(5), 1739–1743 (2013)
Zhang, X.; Pham, H.: An analysis of factors affecting software reliability. J. Syst. Softw. 50(1), 43–56 (2000)
Weyuker, E.J.: Evaluating software complexity measures. IEEE Trans. Softw. Eng. 14(9), 1357–1365 (1988)
Albrecht, A.J.; Gaffney, J.E.: Software function, source lines of code, and development effort prediction: a software science validation. IEEE Trans. Softw. Eng. 6, 639–648 (1983)
Boehm, B.W.: Software Engineering Economics, vol. 197. Prentice-hall, Englewood Cliffs (1981)
Khoshgoftaar, T.M.; Munson, J.C.: Predicting software development errors using software complexity metrics. IEEE J. Sel. Areas Commun. 8(2), 253–261 (1990)
Khan, M.E.; Khan, F.: Importance of software testing in software development life cycle. Int. J. Comput. Sci. Issues (IJCSI) 11(2), 120 (2014)
Banker, R.D.; Datar, S.M.; Kemerer, C.F.; Zweig, D.: Software complexity and maintenance costs. Commun. ACM 36(11), 81–95 (1993)
Banker, R.D.; Davis, G.B.; Slaughter, S.A.: Software development practices, software complexity, and software maintenance performance: a field study. Manag. Sci. 44(4), 433–450 (1998)
Zhu, H.; Hall, P.A.; May, J.H.: Software unit test coverage and adequacy. ACM Comput. Surv. (csur) 29(4), 366–427 (1997)
Sarro, F.; Petrozziello, A.; Harman, M.: Multi-objective software effort estimation. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp. 619–630. IEEE (2016)
Barry, E.J.; Mukhopadhyay, T.; Slaughter, S.A.: Software project duration and effort: an empirical study. Inf. Technol. Manag. 3(1–2), 113–136 (2002)
Morgenshtern, O.; Raz, T.; Dvir, D.: Factors affecting duration and effort estimation errors in software development projects. Inf. Softw. Technol. 49(8), 827–837 (2007)
Subramanian, G.H.; Pendharkar, P.C.; Pai, D.R.: An examination of determinants of software testing and project management effort. J. Comput. Inf. Syst. 57(2), 123–129 (2017)
Harter, D.E.; Krishnan, M.S.; Slaughter, S.A.: Effects of process maturity on quality, cycle time, and effort in software product development. Manag. Sci. 46(4), 451–466 (2000)
Yang, Q.; Li, J.J.; Weiss, D.M.: A survey of coverage-based testing tools. Comput. J. 52(5), 589–597 (2009)
Horgan, J.R.; Mathur, A.P.: Assessing testing tools in research and education. IEEE Softw. 9(3), 61–69 (1992)
Khan, S.A.; Nadeem, A.: A tool for data flow testing using evolutionary approaches (etodf). In: 2013 IEEE 9th International Conference on Emerging Technologies (ICET), pp. 1–6. IEEE (2013)
Huang, C.Y.; Lyu, M.R.: Optimal release time for software systems considering cost, testing-effort, and test efficiency. IEEE Trans. Reliab. 54(4), 583–591 (2005)
Ahmed, B.S.; Zamli, K.Z.: A variable strength interaction test suites generation strategy using particle swarm optimization. J. Syst. Softw. 84(12), 2171–2185 (2011)
Michael, C.C.; McGraw, G.; Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Softw. Eng. 27(12), 1085–1110 (2001)
Yoo, S.; Harman, M.: Test data regeneration: generating new test data from existing test data. Softw. Test. Verif. Reliab. 22(3), 171–201 (2012)
Matinnejad, R.; Nejati, S.; Briand, L.; Bruckmann, T.; Poull, C.: Search-based automated testing of continuous controllers: framework, tool support, and case studies. Inf. Softw. Technol. 57, 705–722 (2015)
Adrion, W.R.; Branstad, M.A.; Cherniavsky, J.C.: Validation, verification, and testing of computer software. ACM Comput. Surv. (CSUR) 14(2), 159–192 (1982)
Noller, J.A.; Mason Jr, R.S.: Automated software testing framework. U.S. Patent No. 7,694,181. U.S. Patent and Trademark Office, Washington, DC (2010)
Tripathy, S.P.; Sahu, P.M.; Kanhar, D.: Optimization of discrete and continuous testsuite using heuristic algorithm: a comparative study. In: 2013 IEEE Conference on Information and Communication Technologies, pp. 841–846. IEEE (2013)
Latiu, G.I.; Cret, O.A.; Vacariu, L.: Automatic test data generation for software path testing using evolutionary algorithms. In: 2012 Third International Conference on Emerging Intelligent Data and Web Technologies, pp. 1–8. IEEE (2012)
Li, J.J.; Weiss, D.; Yee, H.: Code-coverage guided prioritized test generation. Inf. Softw. Technol. 48(12), 1187–1198 (2006)
Xu, D.; Xu, W.; Kent, M.; Thomas, L.; Wang, L.: An automated test generation technique for software quality assurance. IEEE Trans. Reliab. 64(1), 247–268 (2014)
Huang, C.Y.; Kuo, S.Y.: Analysis of incorporating logistic testing-effort function into software reliability modeling. IEEE Trans. Reliab. 51(3), 261–270 (2002)
Yamada, S.; Ohtera, H.; Narihisa, H.: Software reliability growth models with testing-effort. IEEE Trans. Reliab. 35(1), 19–23 (1986)
Huang, C.Y.; Kuo, S.Y.; Lyu, M.R.: An assessment of testing-effort dependent software reliability growth models. IEEE Trans. Reliab. 56(2), 198–211 (2007)
Tian, J.: Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement. Wiley, New York (2005)
Harman, M.; McMinn, P.; De Souza, J.T.; Yoo, S.: Search based software engineering: Techniques, taxonomy, tutorial. In: Empirical Software Engineering and Verification, pp. 1–59. Springer, Berlin (2010)
Lyu, M.R.: Handbook of Software Reliability Engineering, vol. 222. IEEE Computer Society Press, Los Alamitos (1996)
Mathur, A.P.: Performance, effectiveness, and reliability issues in software testing. In:  Proceedings the Fifteenth Annual International Computer Software and Applications Conference, pp. 604–605. IEEE (1991)
Ramler, R.; Wolfmaier, K.: Economic perspectives in test automation: balancing automated and manual testing with opportunity cost. In: Proceedings of the 2006 International Workshop on Automation of Software Test, pp. 85–91. ACM (2006)
Hoffman, D.: Cost benefits analysis of test automation. STAR West, 99 (1999)
Lissitz, R.W.; Green, S.B.: Effect of the number of scale points on reliability: a Monte Carlo approach. J. Appl. Psychol. 60(1), 10 (1975)
Harvey, R.J.; Billings, R.S.; Nilan, K.J.: Confirmatory factor analysis of the job diagnostic survey: good news and bad news. J. Appl. Psychol. 70(3), 461 (1985)
Hinkin, T.R.; Schriesheim, C.A.: Development and application of new scales to measure the French and Raven (1959) bases of social power. J. Appl. Psychol. 74(4), 561 (1989)
Rummel, R.J.: Applied Factor Analysis. Northwestern University Press, Evanston (1988)
Schwab, D.P.: Construct validity in organizational behavior. Res. Organ. Behav. 2, 3–43 (1980)
Hair, J.F.: Multivariate Data Analysis. Pearson Education India, Noida (2006)
Zikmund, W.G.: Business Research Methods: International Edition. Dryden Press, Fort Worth (1994)
Anderson, J.C.; Gerbing, D.W.: Structural equation modeling in practice: a review and recommended two-step approach. Psychol. Bull. 103(3), 411 (1988)
Fornell, C.; Larcker, D.F.: Evaluating structural equation models with unobservable variables and measurement error. J. Mark. Res. 18(1), 39–50 (1981)
Smith, J.B.; Barclay, D.W.: The effects of organizational differences and trust on the effectiveness of selling partner relationships. J. Mark. 61(1), 3–21 (1997)
Appendix: Survey Items
Appendix: Survey Items
1.1 Software Complexity
Most of the softwares that I test are quite complex.
I usually test softwares that have a large number of LOC.
Softwares that are provided to me for testing are generally small in size.
Softwares that are provided to me for testing are not complex.
1.2 Project Duration
I usually get to test softwares that are the part of long-term project.
I feel that long-term projects require more testing effort.
Testing a long-term project is quite challenging for me.
1.3 Testing Tools
Working with testing tools helps me in producing quality results more quickly.
I usually prefer tools for automating the testing process.
I generally require automation tools for testing.
I do not make use of any tools when it comes to testing.
1.4 Automated Test Suite Generation
I feel that test data generation is a crucial part of software development and testing.
I generally perform testing where the test data generation has been automated.
I prefer automated test data generation during testing a software.
Working with manually generated test data is easy for me.
1.5 Software Test Automation
I prefer automated testing over manual testing.
I feel automating the testing process increases the quality of testing.
I normally follow automated testing.
I work with automated tools for testing.
1.6 Software Testing Effort
I feel that testing requires a lot of effort.
I put a lot of effort in testing.
Software testing is not a difficult task for me.
It takes a lot of time to test a software.
1.7 Software Testing Cost
Software testing is a costly process.
I feel that testing software is costly.
I am of the opinion that software testing is the most costly phase in SDLC.
1.8 Software Testing Quality
I feel that the testing results are not up to the mark.
I usually get good testing outputs using automation tools.
I feel that the quality of the test results is perfect.
Rights and permissions
About this article
Cite this article
Khari, M. Empirical Evaluation of Automated Test Suite Generation and Optimization. Arab J Sci Eng 45, 2407–2423 (2020). https://doi.org/10.1007/s13369-019-03996-3
- Software testing
- Software quality
- Software effort
- Software complexity
- Testing cost
- Statistical model