Software Quality Journal

, Volume 22, Issue 4, pp 717–739 | Cite as

Influences on regression testing strategies in agile software development environments

Article

Abstract

Regression testing is a well-established practice in software development, but in recent years it has seen a change of status and emphasis with the increasing popularity of agile methods, which stress the central role of regression testing in maintaining software quality. The objectives of this article are to investigate regression testing strategies in agile development teams and identify the factors that can influence the adoption and implementation of this practice. We have used a mixed methods approach to our research, beginning with an analysis of the literature to identify research themes related to the adoption of regression testing techniques under agile methodologies, from which we developed an analytical framework for the study. This was followed by three exploratory case studies that we used to exercise the main elements of the framework, develop some key themes of interest, and devise a questionnaire for the final stage of the study, an on-line survey to explore the main issues identified in the case studies across different contexts. Within our specific sample, our results suggest that organizational maturity is a key factor in effective regression testing practices and that the adoption of such practices is helped by a coherent testing philosophy and change management processes. We also found that the return on investment in automated regression testing was positive for our respondents and that adopting these practices in the context of agile methods had been a relatively painless process for the organizations in our survey. We conclude that investing in regression testing tools and processes is likely to be beneficial for organizations. However, further work is needed in assessing how organizational culture impacts on the quality process and the financial outcomes for commercial software development organizations.

Keywords

Regression testing strategy Exploratory case study Survey Agile software development 

References

  1. Damm, L.-O., Lundberg, L., & Olsson, D. (2005). Introducing test automation and test-driven development: an experience report. Electronic Notes in Theoretical Computer Science, 116, 3–15.CrossRefGoogle Scholar
  2. Do, H., Elbaum, S., & Rothermel, G. (2004). Infrastructure support for controlled experimentation with software testing and regression testing techniques. In Proceedings of 2004 international symposium on empirical software engineering (ISESE ‘04) (pp. 60–70).Google Scholar
  3. Dustin, E., Rashka, J., & Paul, J. (1999). Automated software testing. Boston: Addison-Wesley.Google Scholar
  4. Elbaum, S., Gable, D., & Rothermel, G. (2001). Understanding and measuring the sources of variation in the prioritization of regression test suites. In Proceedings of seventh international software metrics symposium (METRICS 2001) (pp. 169–179).Google Scholar
  5. Engström, E., & Runeson, P. (2010). A qualitative survey of regression testing practices. In M. Ali Babar, M. Vierimaa & M. Oivo (Eds.), Proceedings 11th international conference on product-focused software process improvement (PROFES). Lecture Notes in Computer Science (Vol. 6156, pp. 3–16). Berlin, Heidelberg: Springer.Google Scholar
  6. Engström, E., Runeson, P., & Skoglund, M. (2010). A systematic review on regression test selection techniques. Information and Software Technology, 52, 14–30.CrossRefGoogle Scholar
  7. Fenton, N. E., & Ohlsson, N. (2000). Quantitative analysis of faults and failures in a complex software system. IEEE Transactions on Software Engineering, 26(8), 797–814.Google Scholar
  8. Gittens, M., Lutfiyya, H., Bauer, M., Godwin, D., Kim, Y.W., & Gupta, P. (2002). An empirical evaluation of system and regression testing. In Proceedings of the 2002 conference of the centre for advanced studies on collaborative research (CASCON ‘02) (p. 3).Google Scholar
  9. Hetzel, W. (1984). The complete guide to software testing. Wellesley, Mass: QED Information Sciences.Google Scholar
  10. Kim, J.-M., Porter, A., & Rothermel, G. (2005). An empirical study of regression test application frequency. Software Testing, Verification and Reliability, 15(4), 257–279.CrossRefGoogle Scholar
  11. Korel, B., & AI-Yami, A. (1998) Automated regression test generation. In Proceedings of ISSTA 98. Clearwater Beach Florida USA.Google Scholar
  12. Leung, H. K. N., & White, L. (1989). Insights into regression testing. In Proceedings of the conference on software maintenance (pp. 60–69).Google Scholar
  13. Loo, P. S., & Tsai, W. K. (1988). Random testing revisited. Information and Software Technology, 30(7), 402–417.CrossRefGoogle Scholar
  14. Martin, R. (2011). Agile software development, principles, patterns, and practices. London: Pearson.Google Scholar
  15. Meszaros, G. (2003). Agile regression testing using record & playback. In Proceedings of the 18th annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications (OOPSLA ‘03) (pp. 353–360). ACM.Google Scholar
  16. Orso, A., Apiwattanapong, T., & Harrold, M. J. (2003). Leveraging field data for impact analysis and regression testing. In Proceedings of the 9th European software engineering conference (pp. 128–137). ACM.Google Scholar
  17. Persson, C., & Yilmaztürk, N. (2004). Establishment of automated regression testing at ABB: Industrial experience report on ‘Avoiding the Pitfalls’. In Proceedings of the 19th IEEE international conference on automated software engineering. Google Scholar
  18. Puleio, M. (2006). How not to do agile testing. Proceedings of Agile Conference, 2006, 307–314.Google Scholar
  19. Runeson, P., Andersson, C., & Host, M. (2003). Test processes in software product evolution—A qualitative survey on the state of practice. Journal of Software Maintenance and Evolution: Research and Practice, 15, 41–59.CrossRefMATHGoogle Scholar
  20. Runeson, P., & Höst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14, 131–164.CrossRefGoogle Scholar
  21. Salama, R. (2011). A regression testing framework for financial time-series databases. An effective combination of fitnesse, scala, and KDB/Q. In Proceedings of the ACM international conference on object oriented programming systems languages and applications (SPLASH ‘11) (pp. 149–154).Google Scholar
  22. Siegel, S. (1996). Object oriented software testing: a hierarchical approach. New York: Wiley.Google Scholar
  23. Srivastava, A., & Thiagarajan, J. (2002). Effectively prioritizing tests in development environment. In Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis (ISSTA ‘02) (pp. 97–106).Google Scholar
  24. Svensson, H., & Host, M. (2005). Introducing an agile process in a software maintenance and evolution organization. In Proceedings of the 9th European conference on software maintenance and reengineering (pp. 256–264).Google Scholar
  25. Talby, D., Keren, A., Hazzan, O., & Dubinsky, Y. (2006). Agile software testing in a large-scale project. IEEE Software, 23(4), 30–37.CrossRefGoogle Scholar
  26. Tsai, W.-T., Poonawala, M., & Suganuma, H. (1998). Regression testing in an industrial environment. Communications of the ACM, 41(5), 81–86.CrossRefGoogle Scholar
  27. Wong, W. E., Horgan, J. R., London, S., & Agrawal, H. (1997). A study of effective regression testing in practice. In Proceedings of the 8th international symposium on software reliability engineering (pp. 264–274).Google Scholar
  28. Yoo, S., & Harman, M. (2012). Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability, 22(2), 67–120.CrossRefGoogle Scholar
  29. Zheng, J., Robinson, B., Williams, L., & Smiley, K. (2006). Applying regression test selection for COTS-based applications. Proceedings of the 28th international conference on Software engineering (ICSE ‘06) (pp. 512–522).Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.School of Engineering and Advanced TechnologyMassey UniversityAucklandNew Zealand
  2. 2.Agile Utilities New Zealand LtdAucklandNew Zealand

Personalised recommendations