Skip to main content
SpringerLink
Log in

Risk-Based Test Case Prioritization by Correlating System Methods and Their Associated Risks

  • Research Article-Computer Engineering and Computer Science
  • Published:
Arabian Journal for Science and Engineering Aims and scope Submit manuscript

Abstract

Regression testing aims to ensure the quality of a software after modification. However, re-executing the entire test suite in regression testing is a time-consuming, costly, and tedious process that often requires additional budget and time. Thus, limited resources always result in early termination and poor quality software. Test case prioritization aims to improve regression testing by re-scheduling the test cases in a manner that could increase the fault detection rate. Risk-based testing has gained popularity in the area of software testing. However, most of the existing methods compute risk values manually, which makes these methods tiresome, laborious, and slow. In this paper, we propose a semi-automatic risk-based test case prioritization approach based on software modification information and methods (functions) invocation relationship. The objective of this research is to make risk-based testing more systematic and flexible by automating the risk assessment process, and find high-risk faults early. The proposed approach utilizes the requirements modification information, complexity, and size of the methods as the risk indicating factors. We applied an automated procedure to extract these risk factors and compute the risk values of the system methods. The proposed approach is empirically evaluated with two software applications with multiple versions in terms of its fault detection rate, both overall and in risky modules. The experimental results suggest that our proposed approach improves test efficiency by spotting defects early overall and even earlier in the high-risk modules than existing state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Gligoric, M.; Negara, S.; Legunsen, O.; Marinov, D.: An empirical evaluation and comparison of manual and automated test selection. In: ASE’14 Proceedings of 29th ACM/IEEE International Conference on Automated Software Engineering, Vasteras, Sweden, pp. 361–372 (2014)

  2. Lin, C.T.; Tang, K.W.; Kapfhammer, G.M.: Test suite reduction methods that decrease regression testing costs by identifying irreplaceable tests. Inf. Softw. Technol. 56(10), 1322–1344 (2014)

    Article  Google Scholar 

  3. Elbaum, S.; Malishevsky, A.G.; Rothermel, G.: Test case prioritization: a family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002)

    Article  Google Scholar 

  4. Khatibsyarbini, M.; Isa, M.A.; Jawawi, D.N.A.; Tumeng, R.: Test case prioritization approaches in regression testing: a systematic literature review. Inf. Softw. Technol. 93, 74–93 (2018)

    Article  Google Scholar 

  5. Tahat, L.; Korel, B.; Koutsogiannakis, G.; Almasri, N.: State-based models in regression test suite prioritization. Softw. Qual. J. 25, 703–742 (2017)

    Article  Google Scholar 

  6. Catal, C.; Mishra, D.: Test case prioritization: a systematic mapping study. Softw. Qual. J. 21(3), 445–478 (2013)

    Article  Google Scholar 

  7. Park, H.; Ryu, H.; Baik, J.: Historical value-based approach for cost-cognizant test case prioritization to improve the effectiveness of regression testing. In: Proceedings of 2nd Second International Conference on Secure System Integration and Reliability Improvement (SSIRI 2008), Yokohama, Japan, pp. 39–46 (2008)

  8. Yoo, S.; Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. 22(2), 67–120 (2012)

    Article  Google Scholar 

  9. Yu, Y.T.; Lau, M.F.: Fault-based test suite prioritization for specification-based testing. Inf. Softw. Technol. 54(2), 179–202 (2012)

    Article  Google Scholar 

  10. Elbaum, S.; Malishevsky, A.; Rothermel, G.: Incorporating varying test costs and fault severities into test case prioritization. In: Proceedings of IEEE 23rd International Conference on Software Engineering (ICSE), Toronto, Ontario, Canada, pp. 329–338 (2001)

  11. Amland, S.: Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study. J. Syst. Softw. 53, 287–295 (2000)

    Article  Google Scholar 

  12. Felderer, M.; Haisjackl, C.; Pekar, V.; Breu, R.: An exploratory study on risk estimation in risk-based testing approaches. In: Winkler, D., Biffl, S., Bergsmann, J. (eds.) Software Quality. Software and Systems Quality in Distributed and Mobile Environments. SWQD 2015. Lecture Notes in Business Information Processing, vol. 200, pp. 32–43. Springer, Cham (2015)

  13. Stallbaum, H.; Metzger, A.; Pohl, K.: An automated technique for risk-based test case generation and prioritization. In: Proceedings of the 3rd International Workshop on Automation of Software Test (AST ’08), Leipzig, Germany, pp. 67–70 (2008)

  14. Hettiarachchi, C.; Do, H.; Choi, B.: Effective regression testing using requirements and risks. In: Proceedings of 8th International Conference on Software Security and Reliability (SERE), San Francisco, CA, USA, pp. 157–166 (2014)

  15. Rothermel, G.; Untch, R.H.; Chengyun, C.; Harrold, M.J.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)

    Article  Google Scholar 

  16. Fu, W.; Yu, H.; Fan, G.; Ji, X.; Pei, X.: A regression test case prioritization algorithm based on program changes and method invocation relationship. In: Proceedings of 24th Asia-Pacific Software Engineering Conference (APSEC), Nanjing, China, pp. 169–178 (2017)

  17. Ammann, P.; Offutt, J.: Introduction to Software Testing. Cambridge University Press, Cambridge (2008)

    Book  Google Scholar 

  18. Jones, J.A.; Harrold, M.J.: Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans. Softw. Eng. 29(3), 195–209 (2003)

    Article  Google Scholar 

  19. Mei, H.; Hao, D.; Zhang, L.; Zhang, L.; Zhou, J.; Rothermel, G.: A static approach to prioritizing JUnit test cases. IEEE Trans. Softw. Eng. 38(6), 1258–1275 (2012)

    Article  Google Scholar 

  20. Nardo, D.D.; Alshahwan, N.; Briand, L.; Labiche, Y.: Coverage-based regression test case selection, minimization and prioritization: a case study on an industrial system. Softw. Test. Verif. Reliab. 25, 371–396 (2015)

    Article  Google Scholar 

  21. Qu, B.; Nie, C.; Xu, B.; Zhang, X.: Test case prioritization for black box testing. In: Proceedings of IEEE 31st Annual International Computer Software Applications Conference (COMPSAC), Beijing, China, pp. 465-474 (2007)

  22. Srikanth, H.; Cashman, M.; Cohen, M.B.: Test case prioritization of build acceptance tests for an enterprise cloud application: an industrial case study. J. Syst. Softw. 119, 122–135 (2016)

    Article  Google Scholar 

  23. Engström, E.; Runeson, P.; Ljung, A.: Improving regression testing transparency and efficiency with history-based prioritization - an industrial case study. In: Proceedings of 4th IEEE International Conference on Software Testing, Verification and Validation, Berlin, Germany, pp. 367-376 (2011)

  24. Yoo, S.; Harman, M.; Tonella, P.; Susi, A.: Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In: Proceedings of 18th International Symposium on Software Testing and Analysis (ISSTA’09), Chicago, IL, USA, pp. 201-212 (2009)

  25. Schwartz, A.; Do, H.: Cost-effective regression testing through adaptive test prioritization strategies. J. Syst. Softw. 115, 61–81 (2016)

    Article  Google Scholar 

  26. Krishnamoorthi, R.; Mary, S.A.S.A.: Factor oriented requirement coverage based system test case prioritization of new and regression test cases. Inf. Softw. Technol. 51(4), 799–808 (2009)

    Article  Google Scholar 

  27. Srikanth, H.; Williams, L.; Osborne, J.: System test case prioritization of new and regression test cases. In: International Symposium on Empirical Software Engineering (ISESE 2005), Noosa Heads, Australia, pp. 64–73 (2005)

  28. Elbaum, S.; Gable, D.; Rothermel, G.: Understanding and measuring the sources of variation in the prioritization of regression test suites. In: Proceedings of 7th International Software Metrics Symposium, London, UK, pp. 169–179 (2002)

  29. Hou, S.S.; Zhang, L.; Xie, T.; Mei, H.; Sun, J.S.: Applying interface-contract mutation in regression testing of component-based software. In: Proceedings of 23rd IEEE International Conference on Software Maintenance, Paris, France, pp. 174–183 (2007)

  30. Lachmann, R.; Schulze, S.; Nieke, M.; Schaefer, I.: System-level test case prioritization using machine learning. In: Proceedings of IEEE 15th International Conference on Machine Learning Applications (ICMLA), Anaheim, CA, USA, pp. 361–368 (2016)

  31. Wang, F.; Yang, S.C.; Yang, Y.L.: Regression testing based on neural networks and program slicing techniques. In: Wang, Y., Li, T. (eds.) Practical Applications of Intelligent Systems. Advances in Intelligent and Soft Computing, vol. 124, pp. 409–418. Springer, Berlin (2011)

    Google Scholar 

  32. Perini, A.; Susi, A.; Avesani, P.: A machine learning approach to software requirements prioritization. IEEE Trans. Softw. Eng. 39(4), 145–461 (2013)

    Article  Google Scholar 

  33. Mirarab, S.; Tahvildari, L.: An empirical study on Bayesian network-based approach for test case prioritization. In: Proceedings of 1st International Conference on Software Testing, Verification, and Validation, Lillehammer, Norway, pp. 278–287 (2008)

  34. Gao, D.; Guo, X.; Zhao, L.: Test case prioritization for regression testing based on ant colony optimization. In: Proceedings of 6th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, pp. 275–279 (2015)

  35. Erdogan, G.; Li, Y.; Kobro, R.; Seehusen, F.; Stølen, K.: Approaches for the combined use of risk analysis and testing: a systematic literature review. Int. J. Softw. Tools Technol. Transf. 16(5), 627–642 (2014)

    Article  Google Scholar 

  36. Xie, X.; Chen, T.Y.; Kuo, F.: A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Trans. Softw. Eng. Methodol. 22(4), 1–40 (2013)

    Article  Google Scholar 

  37. Felderer, M.; Ramler, R.: Integrating risk-based testing in industrial test processes. Softw. Qual. J. 22(3), 543–575 (2014)

    Article  Google Scholar 

  38. Felderer, M.; Schieferdecker, I.: A taxonomy of risk-based testing. Int. J. Softw. Tools Technol. Transf. 16(5), 559–568 (2014)

    Article  Google Scholar 

  39. Yoon, H.; Choi, B.: A test case prioritization based on degree of risk exposure and its empirical study. Int. J. Softw. Eng. Knowl. Eng. 21(2), 191–209 (2011)

    Article  Google Scholar 

  40. Nayak, S.; Kumar, C.; Tripathi, S.: Enhancing efficiency of the test case prioritization technique by improving the rate of fault detection. Arab. J. Sci. Eng. 42(8), 3307–3323 (2017)

    Article  Google Scholar 

  41. Stallbaum, H.; Metzger, A.: Employing requirements metrics for automating early risk assessment. In: Proceedings of MeReP07, Palma de Mallorca, Spain, pp. 1–12 (2007)

  42. Foidl, H.; Felderer, M.: Integrating software quality models into risk-based testing. Softw. Qual. J. 26(2), 809–847 (2016)

    Article  Google Scholar 

  43. Lachmann, R.; Beddig, S.; Lity, S.; Schulze, S.; Schaefer, I.: Risk-based integration testing of software product lines, In: VAMOS’17 Proceedings of the Eleventh International Workshop on Variability Modelling of Software-intensive Systems, Eindhoven, Netherlands, pp. 52–59 (2017)

  44. Yoon, M.; Lee, E.; Song, M.; Choi, B.: A test case prioritization through correlation of requirement and risk. J. Softw. Eng. Appl. 5(10), 823–835 (2012)

    Article  Google Scholar 

  45. Srikanth, H.; Hettiarachchi, C.; Do, H.: Requirements based test prioritization using risk factors: an industrial study. Inf. Softw. Technol. 69, 71–83 (2016)

    Article  Google Scholar 

  46. Hettiarachchi, C.; Do, H.; Choi, B.: Risk-based test case prioritization using a fuzzy expert system. Inf. Softw. Technol. 69, 1–15 (2016)

    Article  Google Scholar 

  47. Wang, Y.; Zhu, Z.; Yang, B.; Guo, F.; Yu, H.: Using reliability risk analysis to prioritize test cases. J. Syst. Softw. 139, 14–31 (2018)

    Article  Google Scholar 

  48. Jahan, H.; Feng, Z.; Mahmud, S.M.H.; Dong, P.: Version specific test case prioritization approach based on artificial neural network. J. Intell. Fuzzy Syst. 36(6), 6181–6194 (2019)

    Article  Google Scholar 

  49. Roubtsov, V.: EMMA: a free Java code coverage tool (2005). http://emma.sourceforge.net/index.html. Accessed 20 Mar 2019

  50. Vanmali, M.; Last, M.; Kandel, A.: Using a neural network in the software testing process. Int. J. Intell. Syst. 17(1), 45–62 (2002)

    Article  Google Scholar 

  51. Shahamiri, S.R.; Wan-Kadir, W.M.N.; Ibrahim, S.; Hashim, S.Z.M.: Artificial neural networks as multi-networks automated test oracle. Autom. Softw. Eng. 19, 303–334 (2012)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science, Sichuan University, Chengdu, 610065, China

    Hosney Jahan & Ziliang Feng

  2. School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China

    S. M. Hasan Mahmud

Authors
  1. Hosney Jahan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ziliang Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. S. M. Hasan Mahmud
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ziliang Feng.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jahan, H., Feng, Z. & Mahmud, S.M.H. Risk-Based Test Case Prioritization by Correlating System Methods and Their Associated Risks. Arab J Sci Eng 45, 6125–6138 (2020). https://doi.org/10.1007/s13369-020-04472-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-020-04472-z

Keywords

Search

Navigation