Skip to main content
SpringerLink
Log in

Enhanced regression testing technique for agile software development and continuous integration strategies

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

To survive in competitive marketplaces, most organizations have adopted agile methodologies to facilitate continuous integration and faster application delivery and rely on regression testing during application development to validate the quality and reliability of the software after changes have been made. Consequently, for large projects with cost and time constraints, it is extremely difficult to determine which test cases to run at the end of each release. In this paper, a test case prioritization and selection approach is proposed to improve the quality of releases. From existing literature, we analyzed prevailing problems and proposed solution relevant to regression testing in agile practices. The proposed approach is based on two phases. First, test cases are prioritized by clustering those test cases that frequently change. In case of a tie, test cases are prioritized based on their respective failure frequencies and coverage criteria. Second, test cases with a higher frequency of failure or coverage criteria are selected. The proposed technique was validated by an empirical study on three industrial subjects. The results show that the method successfully selects an optimal test suite and increases the fault detection rate (i.e., more than 90% in the case of proposed technique and less than 50% in other techniques), which reduces the number of irrelevant test cases and avoids detecting duplicate faults. The results of evaluation metrics illustrate that the proposed technique significantly outperform (i.e., between 91 and 97%) as compared to other existing regression testing techniques (i.e., between 52 and 68%). Therefore, our model enhances the test case prioritization and selection with the ability for earlier and high fault detection. Thus, pruning out irrelevant test cases and redundant faults and enhancing the regression testing process for agile applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Abu Hasan, M., Abdur Rahman, M., & Saeed Siddik, M. (2017). Test case prioritization based on dissimilarity clustering using historical data analysis. Information, Communication and Computing Technology, 750, 269–281. https://doi.org/10.1007/978-981-10-6544-6_25.

    Article  Google Scholar 

  • Agren, S. M., Knauss, E., Heldal, R., Pelliccione, P., Malmqvist, G., & Boden, J. (2018). The manager perspective on requirements impact on automotive systems development speed. 2018 IEEE 26th International Requirements Engineering Conference (RE), 17–28. https://doi.org/10.1109/RE.2018.00-55.

  • Al-Hajjaji, M., Thüm, T., Lochau, M., Meinicke, J., & Saake, G. (2019). Effective product-line testing using similarity-based product prioritization. Software and Systems Modeling, 18(1), 499–521. https://doi.org/10.1007/s10270-016-0569-2.

    Article  Google Scholar 

  • Alkharabsheh, K., Crespo, Y., Manso, E., & Taboada, J. A. (2018). Software design smell detection: a systematic mapping study. Software Quality Journal. https://doi.org/10.1007/s11219-018-9424-8.

  • Almasri, N., Tahat, L., & Korel, B. (2017). Toward automatically quantifying the impact of a change in systems. Software Quality Journal, 25(3), 601–640. https://doi.org/10.1007/s11219-016-9316-8.

    Article  Google Scholar 

  • Aman, H., Nakano, T., Ogasawara, H., & Kawahara, M. (2018). A topic model and test history-based test case recommendation method for regression testing. 2018 IEEE international conference on software testing, verification and validation workshops (ICSTW), 392–397. https://doi.org/10.1109/ICSTW.2018.00079.

  • Anand, R. V., & Dinakaran, M. (2017). Handling stakeholder conflict by agile requirement prioritization using Apriori technique. Computers and Electrical Engineering, 61, 126–136. https://doi.org/10.1016/j.compeleceng.2017.06.022.

    Article  Google Scholar 

  • Anderson, J., Salem, S., & Do, H. (2014). Improving the effectiveness of test suite through mining historical data. Proceedings of the 11th Working Conference on Mining Software Repositories - MSR 2014, 142–151. https://doi.org/10.1145/2597073.2597084.

  • Anita, & Chauhan, N. (2014). A regression test selection technique by optimizing user stories in an agile environment. IEEE International Advance Computing Conference (IACC), 2014, 1454–1458. https://doi.org/10.1109/IAdCC.2014.6779540.

    Article  Google Scholar 

  • Ansari, A., Khan, A., Khan, A., & Mukadam, K. (2016). Optimized regression test using test case prioritization. Procedia Computer Science, 79, 152–160. https://doi.org/10.1016/j.procs.2016.03.020.

    Article  Google Scholar 

  • Arafeen, M. J., & Do, H. (2013). Test case prioritization using requirements-based clustering. 2013 IEEE sixth international conference on software testing, verification and validation, 312–321. https://doi.org/10.1109/ICST.2013.12.

  • Azizi, M., & Do, H. (2018). A collaborative filtering recommender system for test case prioritization in web applications. Proceedings of the 33rd Annual ACM Symposium on Applied Computing - SAC ‘18, 1560–1567. https://doi.org/10.1145/3167132.3167299.

  • Chen, J., Zhu, L., Chen, T. Y., Towey, D., Kuo, F.-C., Huang, R., & Guo, Y. (2018). Test case prioritization for object-oriented software: an adaptive random sequence approach based on clustering. Journal of Systems and Software, 135, 107–125. https://doi.org/10.1016/j.jss.2017.09.031.

    Article  Google Scholar 

  • del Sagrado, J., & del Águila, I. M. (2018). Stability prediction of the software requirements specification. Software Quality Journal, 26(2), 585–605. https://doi.org/10.1007/s11219-017-9362-x.

    Article  Google Scholar 

  • Do, H. (2016). Recent advances in regression testing techniques. In Advances in Computers (Vol. 103, pp. 53–77). https://doi.org/10.1016/bs.adcom.2016.04.004.

  • Elbaum, S., Rothermel, G., & Penix, J. (2014). Techniques for improving regression testing in continuous integration development environments. Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering - FSE 2014, 235–245. https://doi.org/10.1145/2635868.2635910.

  • Felderer, M., & Herrmann, A. (2019). Comprehensibility of system models during test design: a controlled experiment comparing UML activity diagrams and state machines. Software Quality Journal, 27(1), 125–147. https://doi.org/10.1007/s11219-018-9407-9.

    Article  Google Scholar 

  • Fischer, S., Lopez-Herrejon, R. E., & Egyed, A. (2018). Towards a fault-detection benchmark for evaluating software product line testing approaches. Proceedings of the 33rd Annual ACM Symposium on Applied Computing - SAC ‘18, 2034–2041. https://doi.org/10.1145/3167132.3167350.

  • Flemström, D., Potena, P., Sundmark, D., Afzal, W., & Bohlin, M. (2018). Similarity-based prioritization of test case automation. Software Quality Journal, 26(4), 1421–1449. https://doi.org/10.1007/s11219-017-9401-7.

    Article  Google Scholar 

  • Gultepe, E., & Makrehchi, M. (2018). Improving clustering performance using independent component analysis and unsupervised feature learning. Human-centric Computing and Information Sciences, 8(1), 25. https://doi.org/10.1186/s13673-018-0148-3.

    Article  Google Scholar 

  • Gupta, A., Mishra, N., Tripathi, A., Vardhan, M., & Kushwaha, D. S. (2015). An improved history-based test prioritization technique using code coverage. In Advanced Computer and Communication Engineering Technology (Vol. 315, pp. 437–448). https://doi.org/10.1007/978-3-319-07674-4_43.

  • Haghighatkhah, A., Mäntylä, M., Oivo, M., & Kuvaja, P. (2018). Test prioritization in continuous integration environments. Journal of Systems and Software, 146, 80–98. https://doi.org/10.1016/j.jss.2018.08.061.

    Article  Google Scholar 

  • Heck, P., & Zaidman, A. (2018). A systematic literature review on quality criteria for agile requirements specifications. Software Quality Journal, 26(1), 127–160. https://doi.org/10.1007/s11219-016-9336-4.

    Article  Google Scholar 

  • Hettiarachchi, C., Do, H., & Choi, B. (2016). Risk-based test case prioritization using a fuzzy expert system. Information and Software Technology, 69, 1–15. https://doi.org/10.1016/j.infsof.2015.08.008.

    Article  Google Scholar 

  • Horváth, F., Gergely, T., Beszédes, Á., Tengeri, D., Balogh, G., & Gyimóthy, T. (2019). Code coverage differences of Java bytecode and source code instrumentation tools. Software Quality Journal, 27(1), 79–123. https://doi.org/10.1007/s11219-017-9389-z.

    Article  Google Scholar 

  • Huang, Y.-C., Peng, K.-L., & Huang, C.-Y. (2012). A history-based cost-cognizant test case prioritization technique in regression testing. Journal of Systems and Software, 85(3), 626–637. https://doi.org/10.1016/j.jss.2011.09.063.

    Article  MathSciNet  Google Scholar 

  • Kandil, P., Moussa, S., & Badr, N. (2014). Regression testing approach for large-scale systems. IEEE International Symposium on Software Reliability Engineering Workshops, 2014, 132–133. https://doi.org/10.1109/ISSREW.2014.96.

    Article  Google Scholar 

  • Kandil, P., Moussa, S., & Badr, N. (2015). A methodology for regression testing reduction and prioritization of agile releases. 2015 5th international conference on Information & Communication Technology and accessibility (ICTA), 1–6. https://doi.org/10.1109/ICTA.2015.7426903.

  • Kandil, P., Moussa, S., & Badr, N. (2016). Cluster-based test cases prioritization and selection technique for agile regression testing: cluster-based technique for agile regression testing. Journal of Software: Evolution and Process, 29(6), e1794. https://doi.org/10.1002/smr.1794.

    Article  Google Scholar 

  • Knauss, E., Staron, M., Meding, W., Soder, O., Nilsson, A., & Castell, M. (2015). Supporting continuous integration by code-churn based test selection. 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering, 19–25. https://doi.org/10.1109/RCoSE.2015.11.

  • Lachmann, R., Lity, S., Lischke, S., Beddig, S., Schulze, S., & Schaefer, I. (2015). Delta-oriented test case prioritization for integration testing of software product lines. Proceedings of the 19th international conference on software product line - SPLC ‘15, 81–90. https://doi.org/10.1145/2791060.2791073.

  • Li, X., Wong, W. E., Gao, R., Hu, L., & Hosono, S. (2018). Genetic algorithm-based test generation for software product line with the integration of fault localization techniques. Empirical Software Engineering, 23(1), 1–51. https://doi.org/10.1007/s10664-016-9494-9.

    Article  Google Scholar 

  • Lin, C.-T., Chen, C.-D., Tsai, C.-S., & Kapfhammer, G. M. (2013). History-based test case prioritization with software version awareness 2013 18th international conference on engineering of complex computer systems, 171–172. https://doi.org/10.1109/ICECCS.2013.33.

  • Lu, Y., Lou, Y., Cheng, S., Zhang, L., Hao, D., Zhou, Y., & Zhang, L. (2016). How does regression test prioritization perform in real-world software evolution? Proceedings of the 38th International Conference on Software Engineering - ICSE ‘16, 535–546. https://doi.org/10.1145/2884781.2884874.

  • Ma, C., & Provost, J. (2017). A model-based testing framework with reduced set of test cases for programmable controllers. 2017 13th IEEE Conference on Automation Science and Engineering (CASE), 944–949. https://doi.org/10.1109/COASE.2017.8256225.

  • Ma, T., Ali, S., Yue, T., & Elaasar, M. (2019). Testing self-healing cyber-physical systems under uncertainty: a fragility-oriented approach. Software Quality Journal, 27(2), 615–649. https://doi.org/10.1007/s11219-018-9437-3.

    Article  Google Scholar 

  • Magalhães, C., Andrade, J., Perrusi, L., & Mota, A. (2017). Evaluating an automatic text-based test case selection using a non-instrumented code coverage analysis. Proceedings of the 2nd Brazilian Symposium on Systematic and Automated Software Testing - SAST, 1–9. https://doi.org/10.1145/3128473.3128478.

  • Mahali, P., & Mohapatra, D. P. (2018). Model based test case prioritization using UML behavioural diagrams and association rule mining. International Journal of Systems Assurance Engineering and Management, 9(5), 1063–1079. https://doi.org/10.1007/s13198-018-0736-7.

    Article  Google Scholar 

  • Miranda, B., & Bertolino, A. (2017). Scope-aided test prioritization, selection and minimization for software reuse. Journal of Systems and Software, 131, 528–549. https://doi.org/10.1016/j.jss.2016.06.058.

    Article  Google Scholar 

  • Miranda, B., & Bertolino, A. (2018). An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing. Software Quality Journal, 26(4), 1571–1594. https://doi.org/10.1007/s11219-017-9388-0.

    Article  Google Scholar 

  • Ni, C., Liu, W.-S., Chen, X., Gu, Q., Chen, D.-X., & Huang, Q.-G. (2017). A cluster based feature selection method for cross-project software defect prediction. Journal of Computer Science and Technology, 32(6), 1090–1107. https://doi.org/10.1007/s11390-017-1785-0.

    Article  Google Scholar 

  • Noor, T. B., & Hemmati, H. (2015). A similarity-based approach for test case prioritization using historical failure data. 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), 58–68. https://doi.org/10.1109/ISSRE.2015.7381799.

  • Ouni, A., Kessentini, M., ÓCinnéide, M., Sahraoui, H., Deb, K., & Inoue, K. (2017). MORE: a multi-objective refactoring recommendation approach to introducing design patterns and fixing code smells: MORE: a multi-objective refactoring recommendation approach. Journal of Software: Evolution and Process, 29(5), e1843. https://doi.org/10.1002/smr.1843.

    Article  Google Scholar 

  • Ouriques, J. F. S., Cartaxo, E. G., & Machado, P. D. L. (2018). Test case prioritization techniques for model-based testing: a replicated study. Software Quality Journal, 26(4), 1451–1482. https://doi.org/10.1007/s11219-017-9398-y.

    Article  Google Scholar 

  • Özdağoğlu, G., & Kavuncubaşı, E. (2019). Monitoring the software bug-fixing process through the process mining approach. Journal of Software: Evolution and Process, e2162. https://doi.org/10.1002/smr.2162.

  • Panichella, A., Oliveto, R., Penta, M. D., & De Lucia, A. (2015). Improving multi-objective test case selection by injecting diversity in genetic algorithms. IEEE Transactions on Software Engineering, 41(4), 358–383. https://doi.org/10.1109/TSE.2014.2364175.

    Article  Google Scholar 

  • Rosero, R. H., Gómez, O. S., & Rodríguez, G. (2016). 15 years of software regression testing techniques — a survey. International Journal of Software Engineering and Knowledge Engineering, 26(05), 675–689. https://doi.org/10.1142/S0218194016300013.

    Article  Google Scholar 

  • Rosero, R. H., Gomez, O. S., & Rodriguez, G. (2017). Regression testing of database applications under an incremental software development setting. IEEE Access, 5, 18419–18428. https://doi.org/10.1109/ACCESS.2017.2749502.

    Article  Google Scholar 

  • Shin, S. Y., Nejati, S., Sabetzadeh, M., Briand, L. C., & Zimmer, F. (2018). Test case prioritization for acceptance testing of cyber physical systems: a multi-objective search-based approach. Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2018, 49–60. https://doi.org/10.1145/3213846.3213852.

  • Silva, D., Rabelo, R., Campanha, M., Neto, P. S., Oliveira, P. A., & Britto, R. (2016). A hybrid approach for test case prioritization and selection. IEEE Congress on Evolutionary Computation (CEC), 2016, 4508–4515. https://doi.org/10.1109/CEC.2016.7744363.

    Article  Google Scholar 

  • Souto, S., & d’Amorim, M. (2017). Time-space efficient regression testing for configurable systems. ArXiv:1702.03457 [Cs]. Retrieved from http://arxiv.org/abs/1702.03457

  • Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2017). Reinforcement learning for automatic test case prioritization and selection in continuous integration. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2017, 12–22. https://doi.org/10.1145/3092703.3092709.

  • Thangiah, M., & Basri, S. (2016). A preliminary analysis of various testing techniques in agile development - a systematic literature review. 2016 3rd international conference on computer and information sciences (ICCOINS), 600–605. https://doi.org/10.1109/ICCOINS.2016.7783283.

  • Wang, X., & Zeng, H. (2014). Dynamic test case prioritization based on multi-objective. 15th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), 1–6. https://doi.org/10.1109/SNPD.2014.6888744.

  • Wang, X., & Zeng, H. (2016). History-based dynamic test case prioritization for requirement properties in regression testing. Proceedings of the International Workshop on Continuous Software Evolution and Delivery - CSED ‘16, 41–47. https://doi.org/10.1145/2896941.2896949.

  • Wang, X., Zeng, H., Gao, H., Miao, H., & Lin, W. (2019). Location-based test case prioritization for software embedded in mobile devices using the law of gravitation. Mobile Information Systems, 2019, 1–14. https://doi.org/10.1155/2019/9083956.

    Article  Google Scholar 

  • Zhao, D., Lin, H., Ran, L., Han, M., Tian, J., Lu, L., & Xiang, J. (2019). CVSkSA: cross-architecture vulnerability search in firmware based on kNN-SVM and attributed control flow graph. Software Quality Journal. https://doi.org/10.1007/s11219-018-9435-5.

Download references

Funding

The work reported in this paper was supported by the National Natural Science Foundation of China (Grant No. 61672080).

Author information

Authors and Affiliations

  1. University Institute of Information Technology, Pir Mehr Ali Shah Arid Agriculture University, Rawalpindi, Pakistan

    Sadia Ali & Yaser Hafeez

  2. Department of Software Engineering, Foundation University Islamabad, Rawalpindi Campus, Rawalpindi, Pakistan

    Shariq Hussain

  3. School of Reliability and Systems Engineering, Beihang University, Beijing, China

    Shunkun Yang

Authors
  1. Sadia Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yaser Hafeez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shariq Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shunkun Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shunkun Yang.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ali, S., Hafeez, Y., Hussain, S. et al. Enhanced regression testing technique for agile software development and continuous integration strategies. Software Qual J 28, 397–423 (2020). https://doi.org/10.1007/s11219-019-09463-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-019-09463-4

Keywords

Search

Navigation