Skip to main content
SpringerLink
Log in

A taxonomy of risk-based testing

  • Introduction
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

Software testing has often to be done under severe pressure due to limited resources and a challenging time schedule facing the demand to assure the fulfillment of the software requirements. In addition, testing should unveil those software defects that harm the mission-critical functions of the software. Risk-based testing uses risk (re-)assessments to steer all phases of the test process to optimize testing efforts and limit risks of the software-based system. Due to its importance and high practical relevance, several risk-based testing approaches were proposed in academia and industry. This paper presents a taxonomy of risk-based testing providing a framework to understand, categorize, assess, and compare risk-based testing approaches to support their selection and tailoring for specific purposes. The taxonomy is aligned with the consideration of risks in all phases of the test process and consists of the top-level classes risk drivers, risk assessment, and risk-based test process. The taxonomy of risk-based testing has been developed by analyzing the work presented in available publications on risk-based testing. Afterwards, it has been applied to the work on risk-based testing presented in this special section of the International Journal on Software Tools for Technology Transfer.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. The ALARP principle is typically used for safety-critical, but also for mission-critical systems. It says that the residual risk shall be as low as reasonably practical.

References

  1. Amland, S.: Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. J. Syst. Softw. 53(3), 287–295 (2000)

    Article  Google Scholar 

  2. Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press, Cambridge (2008)

    Book  MATH  Google Scholar 

  3. Bach, J.: Heuristic risk-based testing. Softw. Test. Qual. Eng. Mag. 11, 99 (1999)

    Google Scholar 

  4. Bai, X., Kenett, R.S.: Risk-based adaptive group testing of semantic web services. In: 33rd Annual IEEE international computer software and applications conference (COMPSAC’09). vol. 2, pp. 485–490. IEEE (2009)

  5. Bai, X., Kenett, R.S., Yu, W.: Risk assessment and adaptive group testing of semantic web services. Int. J. Softw. Eng. Knowl. Eng. 22(05), 595–620 (2012)

    Article  Google Scholar 

  6. Briand, L.C., Labiche, Y., He, S.: Automating regression test selection based on UML designs. Inf. Softw. Technol. 51(1), 16–30 (2009)

    Article  Google Scholar 

  7. Carrozza, G., Pietrantuono, R., Russo, S.: Dynamic test planning: a study into an industrial context. STTT in this volume (2014)

  8. Casado, R., Tuya, J., Younas, M.: Testing long-lived web services transactions using a risk-based approach. In: 10th international conference on quality software. pp. 337–340. IEEE (2010)

  9. Chen, Y., Probert, R.L., Sims, D.P.: Specification-based regression test selection with risk analysis. In: proceedings of the 2002 conference of the Centre for Advanced Studies on Collaborative research. p. 1. IBM Press (2002)

  10. Erdogan, G., Li, Y., Runde, R.K., Seehusen, F., Stølen, K.: Approaches for the combined use of risk analysis and testing: a systematic literature review. STTT in this volume (2014)

  11. Felderer, M., Beer, A.: Using defect taxonomies to improve the maturity of the system test process: results from an industrial case study. In: software quality. Increasing value in software and systems development, pp. 125–146. Springer (2013)

  12. Felderer, M., Haisjackl, C., Breu, R., Motz, J.: Integrating manual and automatic risk assessment for risk-based testing, pp. 159–180. Software quality. Process automation in software, development (2012)

  13. Felderer, M., Haisjackl, C., Pekar, V., Breu, R.: A risk assessment framework for software testing. In: ISoLA 2014. Springer (2014)

  14. Felderer, M., Ramler, R.: Experiences and challenges of introducing risk-based testing in an industrial project. In: Software quality. Increasing value in software and systems development, pp. 10–29. Springer (2013)

  15. Felderer, M., Ramler, R.: Integrating risk-based testing in industrial test processes. Softw. Qual. J. 22(3), 543–575 (2014)

    Article  Google Scholar 

  16. Felderer, M., Ramler, R.: A multiple case study on risk-based testing in industry. STTT in this volume (2014)

  17. Fredriksen, R., Kristiansen, M., Gran, B.A., Stølen, K., Opperud, T.A., Dimitrakos, T.: The coras framework for a model-based risk management process. In: Anderson, S., Bologna, S., Felici, M. (eds.) SAFECOMP. Lecture Notes in Computer Science, vol. 2434, pp. 94–105. Springer (2002)

  18. Gerrard, P., Thompson, N.: Risk-based e-business testing. Artech House Publishers, (2002)

  19. Goel, A.L.: Software reliability models: assumptions, limitations, and applicability. IEEE Trans. Softw. Eng. 11(12), 1411–1423 (Dec 1985)

  20. Graham, D., Fewster, M.: Experiences of test automation: case studies of software test automation. Addison-Wesley Professional, (2012)

  21. Hosseingholizadeh, A.: A source-based risk analysis approach for software test optimization. In: Computer Engineering and Technology (ICCET), 2010 2nd international conference on. vol. 2, pp. V2601–V2604. IEEE (2010)

  22. Huizinga, D., Kolawa, A.: Automated defect prevention: best practices in software management. Wiley (2007)

  23. IEEE: IEEE Standard for Software and System Test Documentation. IEEE Std 829–2008 (2008)

  24. ISO: ISO 14971: medical devices—application of risk management to medical devices. ISO (2000)

  25. ISO: ISO/IEC/IEEE 29119 Software Testing. http://www.softwaretestingstandard.org/ (2013). Accessed 6 May 2014

  26. ISTQB: Standard glossary of terms used in software testing. version 2.2. Tech. rep., ISTQB (2012)

  27. Jorgensen, M., Boehm, B., Rifkin, S.: Software development effort estimation: formal models or expert judgment? IEEE Softw. 26(2), 14–19 (2009)

    Article  Google Scholar 

  28. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Tech. rep., Technical report, EBSE Technical Report EBSE-2007-01 (2007)

  29. Kloos, J., Hussain, T., Eschbach, R.: Risk-based testing of safety-critical embedded systems driven by fault tree analysis. In: ICSTW 2011. pp. 26–33. IEEE (2011)

  30. Kumar, N., Sosale, D., Konuganti, S.N., Rathi, A.: Enabling the adoption of aspects-testing aspects: a risk model, fault model and patterns. In: proceedings of the 8th ACM international conference on Aspect-oriented software development. pp. 197–206. ACM (2009)

  31. Murthy, K.K., Thakkar, K.R., Laxminarayan, S.: Leveraging risk based testing in enterprise systems security validation. In: first international conference on emerging network intelligence. pp. 111–116. IEEE (2009)

  32. Neubauer, J., Windmüller, S., Steffen, B.: Risk-based testing via active continuous quality control. STTT in this volume (2014)

  33. Radatz, J., Geraci, A., Katki, F.: IEEE standard glossary of software engineering terminology. IEEE Std. 610121990, 121990 (1990)

    Google Scholar 

  34. Ray, M., Mohapatra, D.P.: Risk analysis: a guiding force in the improvement of testing. IET Softw. 7(1), 29–46 (2013)

    Article  Google Scholar 

  35. Redmill, F.: Exploring risk-based testing and its implications. Softw. Test. Verif. Reliab. 14(1), 3–15 (2004)

    Article  Google Scholar 

  36. Redmill, F.: Theory and practice of risk-based testing. Softw. Test. Verif. Reliab. 15(1), 3–20 (2005)

    Article  Google Scholar 

  37. Rosenberg, L., Stapko, R., Gallo, A.: Risk-based object oriented testing. Proceedings of 13th international software/internet quality week-QW 2 (2000)

  38. Schieferdecker, I., Grossmann, J., Schneider, M.: Model-based security testing. Proceedings 7th workshop on model-based testing (2012)

  39. Souza, E., Gusmao, C., Alves, K., Venancio, J., Melo, R.: Measurement and control for risk-based test cases and activities. In: 10th Latin American test workshop. pp. 1–6. IEEE (2009)

  40. Souza, E., Gusmão, C., Venâncio, J.: Risk-based testing: A case study. In: information technology: new generations (ITNG), 2010 seventh international conference on. pp. 1032–1037. IEEE (2010)

  41. Stallbaum, H., Metzger, A.: Employing requirements metrics for automating early risk assessment. Proceedings of MeReP07, Palma de Mallorca, Spain. pp. 1–12 (2007)

  42. Stallbaum, H., Metzger, A., Pohl, K.: An automated technique for risk-based test case generation and prioritization. In: Proceedings of the 3rd international workshop on Automation of software test. pp. 67–70. ACM (2008)

  43. Stallbaum, H., Metzger, A., Pohl, K.: An automated technique for risk-based test case generation and prioritization. In: proceedings of the 3rd international workshop on automation of software test. pp. 67–70. AST ’08, ACM, New York, NY, USA (2008)

  44. Standards Australia/New Zealand: risk management AS/NZS 4360:2004 (2004)

  45. Tran, V., Liu, D.B.: A risk-mitigating model for the development of reliable and maintainable large-scale commercial-off-the-shelf integrated software systems. In: reliability and maintainability symposium. 1997 proceedings, annual. pp. 361–367 (1997)

  46. van Veenendaal, E.: Practical risk-based testing—The PRISMA Approach. UTN Publishers (2012)

  47. Wendland, M.F., Kranz, M., Schieferdecker, I.: A systematic approach to risk-based testing using risk-annotated requirements models. ICSEA 2012, 636–642 (2012)

    Google Scholar 

  48. Windmüller, S., Neubauer, J., Steffen, B., Howar, F., Bauer, O.: Active continuous quality control. In: proceedings of the 16th international ACM sigsoft symposium on component-based software engineering. pp. 111–120. ACM (2013)

  49. Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. 22(2), 67–120 (Mar 2012)

  50. Yoon, H., Choi, B.: A test case prioritization based on degree of risk exposure and its empirical study. Int. J. Softw. Eng. Know. Eng. 21(02), 191–209 (2011)

    Article  Google Scholar 

  51. Zech, P.: Risk-based security testing in cloud computing environments. In: ICST 2011. pp. 411–414. IEEE (2011)

  52. Zech, P., Felderer, M., Breu, R.: Towards risk-driven security testing of service centric systems. In: QSIC. pp. 140–143 (2012)

  53. Zimmermann, F., Eschbach, R., Kloos, J., Bauer, T., et al.: Risk-based statistical testing: A refinement-based approach to the reliability analysis of safety-critical systems. In: EWDC 2009 (2009)

Download references

Acknowledgments

This research was partially funded by the research projects MOBSTECO (FWF P 26194-N15), QE LaB - Living Models for Open Systems (FFG 822740), ITEA2 DIAMONDS (Development and Industrial Application of Multi-Domain-Security Testing Technologies), and EU RASEN (Compositional Risk Assessment and Security Testing of Networked Systems).

Author information

Authors and Affiliations

  1. University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  2. Fraunhofer Institute FOKUS and Freie Universität Berlin, Berlin, Germany

    Ina Schieferdecker

Authors
  1. Michael Felderer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ina Schieferdecker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Felderer.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Felderer, M., Schieferdecker, I. A taxonomy of risk-based testing. Int J Softw Tools Technol Transfer 16, 559–568 (2014). https://doi.org/10.1007/s10009-014-0332-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-014-0332-3

Keywords

Search

Navigation