Skip to main content
  • Research Article - Special Issue - Intelligent Computing and Interdisciplinary Applications
  • Published:

Empirical Evaluation of Automated Test Suite Generation and Optimization

Arabian Journal for Science and Engineering volume 45, pages 2407–2423 (2020)Cite this article

Abstract

Software testing is one of the most labor-intensive and crucial phases of the software development life cycle. It accounts for about 50–60% of the total cost of a project and requires a substantial amount of effort to ensure that the quality of the software is not compromised. Hence, optimizing the process of software testing has been of paramount importance in the IT industry. In recent times, there has been a significant shift toward software test automation to ensure the quality of the software while minimizing the cost and effort. This study proposes a statistical model analyzing the various factors that affect the testing cost, quality and effort by studying the behavioral pattern of professionals employed in the field of testing. The paper studies the factors, project duration, Software Complexity (SC), testing tools and automated test suite generation, while contrasting it with the overall software test automation, with the aim of finding a measure of how much automation affects the cost, quality and effort in practice. The results indicated that automation accounted for a variation of about 25% in each of the factors: cost, quality and effort much more than factors such as project duration, SC and testing tool . Further, the analysis indicated that the effect of SC on cost and effort was extremely low. This suggests that automation improved the quality while reducing the cost and effort irrespective of the complexity of the software being developed.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. Cohen, C.F.; Birkin, S.J.; Garfield, M.J.; Webb, H.W.: Managing conflict in software testing. Commun. ACM 47(1), 76–81 (2004)

    Google Scholar 

  2. Planning, S.: The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology (2002)

  3. Poon, P.L.; Tse, T.H.; Tang, S.F.; Kuo, F.C.: Contributions of tester experience and a checklist guideline to the identification of categories and choices for software testing. Softw. Qual. J. 19(1), 141–163 (2011)

    Google Scholar 

  4. Boehm, B.W.; Brown, J.R.; Lipow, M.: Quantitative evaluation of software quality. In: Proceedings of the 2nd International Conference on Software Engineering, pp. 592–605. IEEE Computer Society Press (1976)

  5. Horgan, J.R.; London, S.; Lyu, M.R.: Achieving software quality with testing coverage measures. Computer 27(9), 60–69 (1994)

    Google Scholar 

  6. Sthamer, H.H.: The automatic generation of software test data using genetic algorithms. Doctoral dissertation, University of Glamorgan (1995)

  7. Ghiduk, A.S.; Girgis, M.R.: Using genetic algorithms and dominance concepts for generating reduced test data. Informatica 34(3), 377–385 (2010)

    Google Scholar 

  8. Prasanna, M.; Sivanandam, S.; Venkatesan, R.; Sundarrajan, R.: A survey on automatic test case generation. Acad. Open Internet J. 15(6) (2005)

  9. Tran, H.; Chechik, M.: Test generation using model checking. In: European Conference on Software Maintenance and Reengineering, CSMR2001 (2001). http://www.cs.toronto.edu/~chechik/courses00/csc2108/projects/4.pdf

  10. Utting, M.; Pretschner, A.; Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verifi. Reliab. 22(5), 297–312 (2012)

    Google Scholar 

  11. McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verifi. Reliab. 14(2), 105–156 (2004)

    Google Scholar 

  12. Harrold, M.J.: Testing: a roadmap. In: Proceedings of the Conference on the Future of Software Engineering, pp. 61–72. ACM (2000)

  13. Bertolino, A.: Software testing research: achievements, challenges, dreams. In: 2007 Future of Software Engineering, pp. 85–103. IEEE Computer Society (2007)

  14. Woodward, M.R.; Hedley, D.; Hennell, M.A.: Experience with path analysis and testing of programs. IEEE Trans. Softw. Eng. 3, 278–286 (1980)

    Google Scholar 

  15. Clarke, L.A.: A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng. 3, 215–222 (1976)

    MathSciNet  Google Scholar 

  16. McCabe, T.J.: A complexity measure. IEEE Trans. Softw. Eng. 4, 308–320 (1976)

    MathSciNet  MATH  Google Scholar 

  17. Prather, R.E.; Myers, J.P.: The path prefix software testing strategy. IEEE Trans. Softw. Eng. 7, 761–766 (1987)

    MATH  Google Scholar 

  18. Korel, B.: Automated software test data generation. IEEE Trans. Softw. Eng. 16(8), 870–879 (1990)

    Google Scholar 

  19. Frhlich, P.; Link, J.: Automated test case optimization from dynamic models. In: ECOOP 2000 Object-Oriented Programming, pp. 472–491. Springer, Berlin (2000)

  20. Gotlieb, A.; Petit, M.: Path-oriented random testing. In: Proceedings of the 1st International Workshop on Random Testing, pp. 28–35. ACM (2006)

  21. Afzal, W.; Torkar, R.; Feldt, R.: A systematic review of search-based testing for non-functional system properties. Inf. Softw. Technol. 51(6), 957–976 (2009)

    Google Scholar 

  22. Harman, M.; Mansouri, S.A.; Zhang, Y.: Search based software engineering: a comprehensive analysis and review of trends techniques and applications. Department of Computer Science, King’s College London, Tech. Rep. TR-09-03, 23 (2009)

  23. Khari, M.; Kumar, P.: An effective meta-heuristic cuckoo search algorithm for test suite optimization. Informatica 41(3), 363–377 (2017)

    MathSciNet  Google Scholar 

  24. Khari, M.; Kumar, P.: Evolutionary computation-based techniques over multiple data sets: an empirical assessment. Arab. J. Sci. Eng. 43(8), 3875–3885 (2018)

    Google Scholar 

  25. Khari, M.; Kumar, P.; Burgos, D.; Crespo, R.G.: Optimized test suites for automated testing using different optimization techniques. Soft Comput. 22(24), 8341–8352 (2018)

    Google Scholar 

  26. Kumar, P.; Khari, M.: An investigating approach for optimization of software test suite. Recent Adv. Commun. Netw. Technol. (Former. Recent Pat. Telecommun.) 6(1), 41–54 (2017)

    Google Scholar 

  27. Sofokleous, A.A.; Andreou, A.S.: Automatic, evolutionary test data generation for dynamic software testing. J. Syst. Softw. 81(11), 1883–1898 (2008)

    Google Scholar 

  28. Alba, E.; Chicano, F.: Observations in using parallel and sequential evolutionary algorithms for automatic software testing. Comput. Oper. Res. 35(10), 3161–3183 (2008)

    MATH  Google Scholar 

  29. Díaz, E.; Tuya, J.; Blanco, R.; Dolado, J.J.: A tabu search algorithm for structural software testing. Comput. Oper. Res. 35(10), 3052–3072 (2008)

    MATH  Google Scholar 

  30. Harman, M.: The current state and future of search based software engineering. In: 2007 Future of Software Engineering, pp. 342–357. IEEE Computer Society (2007)

  31. Li, H.; Lam, C.P.: Software test data generation using ant colony optimization. In: International Conference on Computational Intelligence, pp. 1–4 (2004)

  32. Tahbildar, H.; Kalita, B.: Automated test data generation based on individual constraints and boundary value. Int. J. Comput. Sci. Issues (IJCSI) 7(5), 350 (2010)

    Google Scholar 

  33. Briand, L.; Labiche, Y.: A UML-based approach to system testing. Softw. Syst. Model. 1(1), 10–42 (2002)

    MATH  Google Scholar 

  34. Patil, S.S.: Functional test case generation based on model driven testing using FSM and UML activity diagram. Int. J. Adv. Res. Comput. Sci. 8(5), 1527–1530 (2017)

    Google Scholar 

  35. Ferrari, F.C.; Nakagawa, E.Y.; Rashid, A.; Maldonado, J.C.: Automating the mutation testing of aspect-oriented Java programs. In: Proceedings of the 5th Workshop on Automation of Software Test, pp. 51–58. ACM (2010)

  36. Guderlei, R.; Just, R.; Schneckenburger, C.; Schweiggert, F.: Benchmarking testing strategies with tools from mutation analysis. In: 2008 IEEE International Conference on Software Testing Verification and Validation Workshop, pp. 360–364. IEEE (2008)

  37. Qu, X.; Cohen, M.B.; Rothermel, G.: Configuration-aware regression testing: an empirical study of sampling and prioritization. In: Proceedings of the 2008 International Symposium on Software Testing and Analysis, pp. 75–86. ACM (2008)

  38. Saha, G.K.: Understanding software testing concepts. Ubiquity 2008(February), 2 (2008)

    Google Scholar 

  39. Ali, S.; Briand, L.C.; Hemmati, H.; Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36(6), 742–762 (2009)

    Google Scholar 

  40. Devaraj, D.; Roselyn, J.P.: Genetic algorithm based reactive power dispatch for voltage stability improvement. Int. J. Electr. Power Energy Syst. 32(10), 1151–1156 (2010)

    Google Scholar 

  41. Harman, M.; McMinn, P.: A theoretical and empirical study of search-based testing: local, global, and hybrid search. IEEE Trans. Softw. Eng. 36(2), 226–247 (2009)

    Google Scholar 

  42. Hermadi, I.; Lokan, C.; Sarker, R.: Genetic algorithm based path testing: challenges and key parameters. In: 2010 Second World Congress on Software Engineering, vol. 2, pp. 241–244 . IEEE (2010)

  43. Michaelis, M.: Boon and bane of GUI test automation. Test. Exp. 17, 25–28 (2008)

    Google Scholar 

  44. Kaur, H.; Gupta, G.: Comparative study of automated testing tools: selenium, quick test professional and testcomplete. Int. J. Eng. Res. Appl. 3(5), 1739–1743 (2013)

    Google Scholar 

  45. Zhang, X.; Pham, H.: An analysis of factors affecting software reliability. J. Syst. Softw. 50(1), 43–56 (2000)

    Google Scholar 

  46. Weyuker, E.J.: Evaluating software complexity measures. IEEE Trans. Softw. Eng. 14(9), 1357–1365 (1988)

    MathSciNet  Google Scholar 

  47. Albrecht, A.J.; Gaffney, J.E.: Software function, source lines of code, and development effort prediction: a software science validation. IEEE Trans. Softw. Eng. 6, 639–648 (1983)

    Google Scholar 

  48. Boehm, B.W.: Software Engineering Economics, vol. 197. Prentice-hall, Englewood Cliffs (1981)

    MATH  Google Scholar 

  49. Khoshgoftaar, T.M.; Munson, J.C.: Predicting software development errors using software complexity metrics. IEEE J. Sel. Areas Commun. 8(2), 253–261 (1990)

    Google Scholar 

  50. Khan, M.E.; Khan, F.: Importance of software testing in software development life cycle. Int. J. Comput. Sci. Issues (IJCSI) 11(2), 120 (2014)

    MathSciNet  Google Scholar 

  51. Banker, R.D.; Datar, S.M.; Kemerer, C.F.; Zweig, D.: Software complexity and maintenance costs. Commun. ACM 36(11), 81–95 (1993)

    Google Scholar 

  52. Banker, R.D.; Davis, G.B.; Slaughter, S.A.: Software development practices, software complexity, and software maintenance performance: a field study. Manag. Sci. 44(4), 433–450 (1998)

    MATH  Google Scholar 

  53. Zhu, H.; Hall, P.A.; May, J.H.: Software unit test coverage and adequacy. ACM Comput. Surv. (csur) 29(4), 366–427 (1997)

    Google Scholar 

  54. Sarro, F.; Petrozziello, A.; Harman, M.: Multi-objective software effort estimation. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp. 619–630. IEEE (2016)

  55. Barry, E.J.; Mukhopadhyay, T.; Slaughter, S.A.: Software project duration and effort: an empirical study. Inf. Technol. Manag. 3(1–2), 113–136 (2002)

    Google Scholar 

  56. Morgenshtern, O.; Raz, T.; Dvir, D.: Factors affecting duration and effort estimation errors in software development projects. Inf. Softw. Technol. 49(8), 827–837 (2007)

    Google Scholar 

  57. Subramanian, G.H.; Pendharkar, P.C.; Pai, D.R.: An examination of determinants of software testing and project management effort. J. Comput. Inf. Syst. 57(2), 123–129 (2017)

    Google Scholar 

  58. Harter, D.E.; Krishnan, M.S.; Slaughter, S.A.: Effects of process maturity on quality, cycle time, and effort in software product development. Manag. Sci. 46(4), 451–466 (2000)

    Google Scholar 

  59. Yang, Q.; Li, J.J.; Weiss, D.M.: A survey of coverage-based testing tools. Comput. J. 52(5), 589–597 (2009)

    Google Scholar 

  60. Horgan, J.R.; Mathur, A.P.: Assessing testing tools in research and education. IEEE Softw. 9(3), 61–69 (1992)

    Google Scholar 

  61. Khan, S.A.; Nadeem, A.: A tool for data flow testing using evolutionary approaches (etodf). In: 2013 IEEE 9th International Conference on Emerging Technologies (ICET), pp. 1–6. IEEE (2013)

  62. Huang, C.Y.; Lyu, M.R.: Optimal release time for software systems considering cost, testing-effort, and test efficiency. IEEE Trans. Reliab. 54(4), 583–591 (2005)

    Google Scholar 

  63. Ahmed, B.S.; Zamli, K.Z.: A variable strength interaction test suites generation strategy using particle swarm optimization. J. Syst. Softw. 84(12), 2171–2185 (2011)

    Google Scholar 

  64. Michael, C.C.; McGraw, G.; Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Softw. Eng. 27(12), 1085–1110 (2001)

    Google Scholar 

  65. Yoo, S.; Harman, M.: Test data regeneration: generating new test data from existing test data. Softw. Test. Verif. Reliab. 22(3), 171–201 (2012)

    Google Scholar 

  66. Matinnejad, R.; Nejati, S.; Briand, L.; Bruckmann, T.; Poull, C.: Search-based automated testing of continuous controllers: framework, tool support, and case studies. Inf. Softw. Technol. 57, 705–722 (2015)

    Google Scholar 

  67. Adrion, W.R.; Branstad, M.A.; Cherniavsky, J.C.: Validation, verification, and testing of computer software. ACM Comput. Surv. (CSUR) 14(2), 159–192 (1982)

    Google Scholar 

  68. Noller, J.A.; Mason Jr, R.S.: Automated software testing framework. U.S. Patent No. 7,694,181. U.S. Patent and Trademark Office, Washington, DC (2010)

  69. Tripathy, S.P.; Sahu, P.M.; Kanhar, D.: Optimization of discrete and continuous testsuite using heuristic algorithm: a comparative study. In: 2013 IEEE Conference on Information and Communication Technologies, pp. 841–846. IEEE (2013)

  70. Latiu, G.I.; Cret, O.A.; Vacariu, L.: Automatic test data generation for software path testing using evolutionary algorithms. In: 2012 Third International Conference on Emerging Intelligent Data and Web Technologies, pp. 1–8. IEEE (2012)

  71. Li, J.J.; Weiss, D.; Yee, H.: Code-coverage guided prioritized test generation. Inf. Softw. Technol. 48(12), 1187–1198 (2006)

    Google Scholar 

  72. Xu, D.; Xu, W.; Kent, M.; Thomas, L.; Wang, L.: An automated test generation technique for software quality assurance. IEEE Trans. Reliab. 64(1), 247–268 (2014)

    Google Scholar 

  73. Huang, C.Y.; Kuo, S.Y.: Analysis of incorporating logistic testing-effort function into software reliability modeling. IEEE Trans. Reliab. 51(3), 261–270 (2002)

    MathSciNet  Google Scholar 

  74. Yamada, S.; Ohtera, H.; Narihisa, H.: Software reliability growth models with testing-effort. IEEE Trans. Reliab. 35(1), 19–23 (1986)

    Google Scholar 

  75. Huang, C.Y.; Kuo, S.Y.; Lyu, M.R.: An assessment of testing-effort dependent software reliability growth models. IEEE Trans. Reliab. 56(2), 198–211 (2007)

    Google Scholar 

  76. Tian, J.: Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement. Wiley, New York (2005)

    Google Scholar 

  77. Harman, M.; McMinn, P.; De Souza, J.T.; Yoo, S.: Search based software engineering: Techniques, taxonomy, tutorial. In: Empirical Software Engineering and Verification, pp. 1–59. Springer, Berlin (2010)

  78. Lyu, M.R.: Handbook of Software Reliability Engineering, vol. 222. IEEE Computer Society Press, Los Alamitos (1996)

    Google Scholar 

  79. Mathur, A.P.: Performance, effectiveness, and reliability issues in software testing. In: [1991] Proceedings the Fifteenth Annual International Computer Software and Applications Conference, pp. 604–605. IEEE (1991)

  80. Ramler, R.; Wolfmaier, K.: Economic perspectives in test automation: balancing automated and manual testing with opportunity cost. In: Proceedings of the 2006 International Workshop on Automation of Software Test, pp. 85–91. ACM (2006)

  81. Hoffman, D.: Cost benefits analysis of test automation. STAR West, 99 (1999)

  82. Lissitz, R.W.; Green, S.B.: Effect of the number of scale points on reliability: a Monte Carlo approach. J. Appl. Psychol. 60(1), 10 (1975)

    Google Scholar 

  83. Harvey, R.J.; Billings, R.S.; Nilan, K.J.: Confirmatory factor analysis of the job diagnostic survey: good news and bad news. J. Appl. Psychol. 70(3), 461 (1985)

    Google Scholar 

  84. Hinkin, T.R.; Schriesheim, C.A.: Development and application of new scales to measure the French and Raven (1959) bases of social power. J. Appl. Psychol. 74(4), 561 (1989)

    Google Scholar 

  85. Rummel, R.J.: Applied Factor Analysis. Northwestern University Press, Evanston (1988)

    MATH  Google Scholar 

  86. Schwab, D.P.: Construct validity in organizational behavior. Res. Organ. Behav. 2, 3–43 (1980)

    Google Scholar 

  87. Hair, J.F.: Multivariate Data Analysis. Pearson Education India, Noida (2006)

    Google Scholar 

  88. Zikmund, W.G.: Business Research Methods: International Edition. Dryden Press, Fort Worth (1994)

    Google Scholar 

  89. Anderson, J.C.; Gerbing, D.W.: Structural equation modeling in practice: a review and recommended two-step approach. Psychol. Bull. 103(3), 411 (1988)

    Google Scholar 

  90. Fornell, C.; Larcker, D.F.: Evaluating structural equation models with unobservable variables and measurement error. J. Mark. Res. 18(1), 39–50 (1981)

    Google Scholar 

  91. Smith, J.B.; Barclay, D.W.: The effects of organizational differences and trust on the effectiveness of selling partner relationships. J. Mark. 61(1), 3–21 (1997)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Ambedkar Institute of Advanced Communication Technology and Research, Geeta Colony, New Delhi, Delhi, 110032, India

    Manju Khari

Authors
  1. Manju Khari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manju Khari.

Appendix: Survey Items

Appendix: Survey Items

1.1 Software Complexity

  1. SC1

    Most of the softwares that I test are quite complex.

  2. SC2

    I usually test softwares that have a large number of LOC.

  3. SC3

    Softwares that are provided to me for testing are generally small in size.

  4. SC4

    Softwares that are provided to me for testing are not complex.

1.2 Project Duration

  1. PD1

    I usually get to test softwares that are the part of long-term project.

  2. PD2

    I feel that long-term projects require more testing effort.

  3. PD3

    Testing a long-term project is quite challenging for me.

1.3 Testing Tools

  1. TT1

    Working with testing tools helps me in producing quality results more quickly.

  2. TT2

    I usually prefer tools for automating the testing process.

  3. TT3

    I generally require automation tools for testing.

  4. TT4

    I do not make use of any tools when it comes to testing.

1.4 Automated Test Suite Generation

  1. ATG1

    I feel that test data generation is a crucial part of software development and testing.

  2. ATG2

    I generally perform testing where the test data generation has been automated.

  3. ATG3

    I prefer automated test data generation during testing a software.

  4. ATG4

    Working with manually generated test data is easy for me.

1.5 Software Test Automation

  1. STA1

    I prefer automated testing over manual testing.

  2. STA2

    I feel automating the testing process increases the quality of testing.

  3. STA3

    I normally follow automated testing.

  4. STA4

    I work with automated tools for testing.

1.6 Software Testing Effort

  1. STE1

    I feel that testing requires a lot of effort.

  2. STE2

    I put a lot of effort in testing.

  3. STE3

    Software testing is not a difficult task for me.

  4. STE4

    It takes a lot of time to test a software.

1.7 Software Testing Cost

  1. STC1

    Software testing is a costly process.

  2. STC2

    I feel that testing software is costly.

  3. STC3

    I am of the opinion that software testing is the most costly phase in SDLC.

1.8 Software Testing Quality

  1. STQ1

    I feel that the testing results are not up to the mark.

  2. STQ2

    I usually get good testing outputs using automation tools.

  3. STQ3

    I feel that the quality of the test results is perfect.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Khari, M. Empirical Evaluation of Automated Test Suite Generation and Optimization. Arab J Sci Eng 45, 2407–2423 (2020). https://doi.org/10.1007/s13369-019-03996-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-019-03996-3

Keywords

  • Software testing
  • Optimization
  • Software quality
  • Software effort
  • Software complexity
  • Testing cost
  • Statistical model