Approaches for the combined use of risk analysis and testing: a systematic literature review

  • Gencer Erdogan
  • Yan Li
  • Ragnhild Kobro Runde
  • Fredrik Seehusen
  • Ketil Stølen
RBT

Abstract

Risk analysis and testing are conducted for different purposes. Risk analysis and testing nevertheless involve processes that may be combined to the benefit of both. We may use testing to support risk analysis and risk analysis to support testing. This paper surveys literature on the combined use of risk analysis and testing. First, the existing approaches are identified through a systematic literature review. The identified approaches are then classified and discussed with respect to main goal, context of use and maturity level. The survey highlights the need for more structure and rigor in the definition and presentation of approaches. Evaluations are missing in most cases. The paper may serve as a basis for examining approaches for the combined use of risk analysis and testing, or as a resource for identifying the adequate approach to use.

Keywords

Risk-based testing Test-based risk analysis Literature survey 

Notes

Acknowledgments

This work has been conducted as a part of the DIAMONDS (201579/S10) project funded by the Research Council of Norway, as well as a part of the NESSoS network of excellence and the RASEN project funded by the European Commission within the 7th Framework Programme.

References

  1. 1.
    Alam, M., Khan, A.I.: Risk-based testing techniques: a perspective study. Int. J. Comput. Appl. 65, 33–41 (2013)Google Scholar
  2. 2.
    Amland, S.: Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study. J. Syst. Softw. 53, 287–295 (2000)CrossRefGoogle Scholar
  3. 3.
    Bai, X., Kenett, R.S.: Risk-based adaptive group testing of semantic web services. In: Proceeding of the 33rd Annual IEEE International Computer Software and Applications Conference (COMPSAC’09), vol. 2, pp. 485–490. IEEE, New York (2009)Google Scholar
  4. 4.
    Bai, X., Kennett, R.S., Yu, W.: Risk assessment and adaptive group testing of semantic web services. Int. J. Softw. Eng. Knowl. Eng. 595–620 (2012)Google Scholar
  5. 5.
    Casado, R., Tuya, J., Younas, M.: Testing long-lived web services transactions using a risk-based approach. In: Proceeding of the 10th International Conference on Quality Software (QSIC’10), pp. 337–340. IEEE, New York (2010)Google Scholar
  6. 6.
    Casado, R., Tuya, J., Younas, M.: A framework to test advanced web services transactions. In: Proceeding of the 4th International Conference on Software Testing, Verification and Validation (ICST’11), pp 443–446. IEEE, New York (2011)Google Scholar
  7. 7.
    Chen, Y., Probert, R.L.: A risk-based regression test selection strategy. In: Proceeding of the 14th IEEE International Symposium on Software Reliability Engineering (ISSRE’03), Fast Abstract, pp. 305–306. Chillarege Press (2003)Google Scholar
  8. 8.
    Chen, Y., Probert, R.L., Sims, D.P.: Specification-based regression test selection with risk analysis. In: Proceeding of the 2002 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON’02), pages 1–14. IBM Press, USA (2002)Google Scholar
  9. 9.
    Entin, V., Winder, M., Zhang, B., Christmann, S.: Introducing model-based testing in an industrial scrum project. In: Proceeding of the Seventh International Workshop on Automation of Software Test (AST’12), pp. 43–49. IEEE, New York (2012)Google Scholar
  10. 10.
    Felderer, M., Haisjackl, C., Breu, R., Motz, J.: Integrating manual and automatic risk assessment for risk-based testing. In: Proceeding of the 4th International Conference on Software Quality (SWQD’12), vol. 94 of LNBIP, pp. 159–180. Springer, Berlin (2012)Google Scholar
  11. 11.
    Felderer, M., Ramler, R.: Experiences and challenges of introducing risk-based testing in an industrial project. In: Proceeding of the Fifth International Conference on Software Quality (SWQD’13), vol. 133 of LNBIP, pp. 10–29. Springer, New York (2013)Google Scholar
  12. 12.
    Gleirscher, M.: Hazard-based selection of test cases. In: Proceeding if the Sixth International Workshop on Automation of Software Test (AST’11), pp. 64–70. ACM, New York (2011)Google Scholar
  13. 13.
    Gleirscher, M.: Hazard analysis of technical systems. In: Proceeding of the Fifth International Conference on Software Quality (SWQD’13), vol. 133 of LNBIP, pp. 104–124. Springer, Berlin (2013)Google Scholar
  14. 14.
    Hosseingholizadeh, A.: A source-based risk analysis approach for software test optimization. In: Proceeding of the Second International Conference on Computer Engineering and Technology (ICCET’10), vol. 2, pp. 601–604. IEEE, New York (2010)Google Scholar
  15. 15.
    International Standards Organization. ISO 31000:2009(E), Risk management—Principles and guidelines (2009)Google Scholar
  16. 16.
    International Standards Organization. ISO 29119 Software and system engineering—Software Testing-Part 2 : Test process (draft) (2012)Google Scholar
  17. 17.
    Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007–001 (2007)Google Scholar
  18. 18.
    Kloos, J., Hussain, T., Eschbach, R.: Risk-based testing of safety-critical embedded systems driven by Fault Tree Analysis. In: Proceeding of the Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW’11), pp. 26–33. IEEE, New York (2011)Google Scholar
  19. 19.
    Kumar, N., Sosale, D., Konuganti, S.N., Rathi, A.: Enabling the adoption of aspects-testing aspects: A risk model, fault model and patterns. In: Proceeding of the Eighth ACM International Conference on Aspect-Oriented Software Development (AOSD’09), pp. 197–206. ACM, New York (2009)Google Scholar
  20. 20.
    Murthy, K.K., Thakkar, K.R., Laxminarayan, S.: Leveraging risk based testing in enterprise systems security validation. In: Proceeding of the First International Conference on Emerging Network Intelligence (EMERGING’09), pp. 111–116. IEEE, New York (2009)Google Scholar
  21. 21.
    Nazier, R., Bauer, T.: Automated risk-based testing by integrating safety analysis information into system behavior models. In: Proceeding of the 23rd International Symposium on Software Reliability Engineering Workshops (ISSREW’12), pp 213–218. IEEE, New York (2012)Google Scholar
  22. 22.
    Ray, M., Mohapatra, D.P.: Risk analysis: a guiding force in the improvement of testing. IET Softw. 7, 29–46 (2013)CrossRefGoogle Scholar
  23. 23.
    Redmill, F.: Exploring risk-based testing and its implications. Softw. Test. Verif. Reliab. 14, 3–15 (2004)CrossRefGoogle Scholar
  24. 24.
    Redmill, F.: Theory and practice of risk-based testing. Softw. Test. Verif. Reliab. 15, 3–20 (2005)CrossRefGoogle Scholar
  25. 25.
    Rosenberg, L., Stapko, R., Gallo, A.: Risk-based object oriented testing. In: Proceeding of the 24th Annual Software Engineering Workshop, pp. 1–6. NASA, Software Engineering Laboratory, (1999)Google Scholar
  26. 26.
    Schneidewind, N.F.: Risk-driven software testing and reliability. Int. J. Reliab. Qual. Saf. Eng. 14, 99–132 (2007)Google Scholar
  27. 27.
    Souza, E., Gusmão, C., Alves, K., Venâncio, J., Melo, R.: Measurement and control for risk-based test cases and activities. In: Proceeding of the 10th Latin American Test Workshop (LATW’09), pp 1–6. IEEE, New York (2009)Google Scholar
  28. 28.
    Souza, E., Gusmão, C., Venâncio, J.: Risk-based testing: A case study. In: Proceeding of the Seventh International Conference on Information Technology: New Generations (ITNG’10), pp. 1032–1037. IEEE, New York (2010)Google Scholar
  29. 29.
    Stallbaum, H., Metzger, A., Pohl, K.: An automated technique for risk-based test case generation and prioritization. In: Proceeding of the Third International Workshop on Automation of Software Test (AST’08), pp. 67–70. ACM, New York (2008)Google Scholar
  30. 30.
    Sulaman, S.M., Weyns, K., Höst, M.: A review of research on risk analysis methods for it systems. In: Proceeding of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE’13), pp. 86–96 (2013)Google Scholar
  31. 31.
    Wendland, M.-F., Kranz, M., Schieferdecker, I.: A systematic approach to risk-based testing using risk-annotated requirements models. In: Proceeding of the Seventh International Conference on Software Engineering Advances (ICSEA’12), pp. 636–642. IARA (2012)Google Scholar
  32. 32.
    Wong, W.E., Qi, Y., Cooper, K.: Source code-based software risk assessing. In: Proceeding of the 2005 ACM Symposium on Applied Computing (SAC’05), pp. 1485–1490. ACM, New York (2005)Google Scholar
  33. 33.
    Xu, D., Tu, M., Sandford, M., Thomas, L., Woodraska, D., Xu, W.: Automated security test generation with formal threat models. IEEE Trans. Dependable Secure Comput. 9, 526–540 (2012)CrossRefGoogle Scholar
  34. 34.
    Yoon, H., Choi, B.: A test case prioritization based on degree of risk exposure and its empirical study. Int. J. Softw. Eng. Knowl. Eng. 21, 191–209 (2011)CrossRefGoogle Scholar
  35. 35.
    Zech, P.: Risk-based security testing in cloud computing environments. In: Proceeding of the Fourth International Conference on Software Testing, Verification and Validation (ICST’11), pp. 411–414. IEEE, New York (2011)Google Scholar
  36. 36.
    Zech, P., Felderer, M., Breu, R.: Towards a model based security testing approach of cloud computing environments. In: Proceeding of the Sixth International Conference on Software Security and Reliability Companion (SERE-C’12), pp. 47–56. IEEE, New York (2012)Google Scholar
  37. 37.
    Zimmermann, F., Eschbach, R., Kloos, J., Bauer, T.: Risk-based statistical testing: A refinement-based appraoch to the reliability analysis of safety-critical systems. In: Proceeding of the 12th European Workshop on Dependable Computing (EWDC’09), pp. 1–8 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Gencer Erdogan
    • 1
    • 2
  • Yan Li
    • 1
    • 2
  • Ragnhild Kobro Runde
    • 2
  • Fredrik Seehusen
    • 1
  • Ketil Stølen
    • 1
    • 2
  1. 1.SINTEF ICTOsloNorway
  2. 2.Department of InformaticsUniversity of OsloOsloNorway

Personalised recommendations