A multiple case study on risk-based testing in industry

  • Michael FeldererEmail author
  • Rudolf Ramler


In many development projects, testing has to be conducted under severe pressure due to limited resources and a challenging time schedule. Risk-based testing, which utilizes identified risks of the system for testing purposes, has a high potential to improve testing as it helps to optimize the allocation of resources and provides decision support for management. But for many organizations, the integration of a risk-based approach into established testing activities is a challenging task, and there are several options to do so. In this article, we analyze how risk is defined, assessed, and applied to support and improve testing activities in projects, products, and processes. We investigate these questions empirically by a multiple case study of currently applied risk-based testing activities in industry. The case study is based on three cases from different backgrounds, i.e., a test project in context of the extension of a large Web-based information system, product testing of a measurement and diagnostic equipment for the electrical power industry, as well as a test process of a system integrator of telecommunication solutions. By analyzing and comparing these different industrial cases, we draw conclusions on the state of risk-based testing and discuss possible improvements.


Risk-based testing Case study research Multiple case study Test process improvement Test management Risk management Software testing Software quality 



This work has been supported by the COMET Competence Center program of the Austrian Research Promotion Agency (FFG), the project QE LaB Living Models for Open Systems ( funded by the Austrian Federal Ministry of Economics (Bundesministerium für Wirtschaft und Arbeit), the project MOBSTECO funded by the Austrian Science Fund (FWF) as well as the competence network Softnet Austria ( funded by the Austrian Federal Ministry of Economics (Bundesministerium für Wirtschaft und Arbeit), the province of Styria, the Steirische Wirtschaftsförderungsgesellschaft mbH (SFG), and the city of Viennas Center for Innovation and Technology (ZIT).


  1. 1.
    Wendland, M.F., Kranz, M., Schieferdecker, I.: A systematic approach to risk-based testing using risk-annotated requirements models. In: ICSEA 2012. The Seventh International Conference on Software Engineering Advances, pp. 636–642 (2012)Google Scholar
  2. 2.
    Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Eng. 14(2), 131–164 (2009)CrossRefGoogle Scholar
  3. 3.
    Yin, R.K.: Case study research: design and methods, vol. 5. Sage (2009)Google Scholar
  4. 4.
    Bach, J.: Heuristic risk-based testing. Softw. Test. Qual. Eng. Mag. 11, 99 (1999)Google Scholar
  5. 5.
    Amland, S.: Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study. J. Syst. Softw. 53(3), 287–295 (2000)CrossRefGoogle Scholar
  6. 6.
    Redmill, F.: Theory and practice of risk-based testing. Softw. Test. Verif. Reliab. 15(1), 3–20 (2005)Google Scholar
  7. 7.
    van Veenendaal, E.: Practical risk-based testing: the PRISMA approach. UTN, Cambridge (2012)Google Scholar
  8. 8.
    Felderer, M., Haisjackl, C., Breu, R., Motz, J.: Integrating manual and automatic risk assessment for risk-based testing. Software Quality. Process Automation in Software Development, pp. 159–180 (2012)Google Scholar
  9. 9.
    ISO: ISO/IEC/IEEE 29119 Software Testing (2013). Available at Accessed May 6 2014
  10. 10.
    van Veenendaal, E. (ed.): Standard glossary of terms used in software testing, version 2.2. Technical report, International Software Testing Qualifications Board, Glossary Working Party (2012)Google Scholar
  11. 11.
    ISO: ISO/IEC/IEEE 24765:2010 System and software engineering—Vocabulary (2010)Google Scholar
  12. 12.
    Cavano, J., McCall, J.: A framework for the measurement of software quality. ACM SIGMETRICS Perform. Eval. Rev. 7(3–4), 133–139 (1978)Google Scholar
  13. 13.
    Redmill, F.: Exploring risk-based testing and its implications. Softw. Test. Verif. Reliab. 14(1), 3–15 (2004)Google Scholar
  14. 14.
    Standards Australia/New Zealand: Risk Management AS/NZS 4360, 2004 (2004)Google Scholar
  15. 15.
    Karolak, D., Karolak, N.: Software Engineering Risk Management: A Just-in-Time Approach. IEEE Computer Society Press, Los Alamitos (1995)Google Scholar
  16. 16.
    Chen, Y., Probert, R.L., Sims, D.P.: Specification-based regression test selection with risk analysis. In: Proceedings of the 2002 Conference of the Centre for Advanced Studies on Collaborative Research. IBM Press (2002)Google Scholar
  17. 17.
    Stallbaum, H., Metzger, A.: Employing requirements metrics for automating early risk assessment. In: Proceedings of MeReP07, Palma de Mallorca, Spain, pp. 1–12 (2007)Google Scholar
  18. 18.
    Stallbaum, H., Metzger, A., Pohl, K.: An automated technique for risk-based test case generation and prioritization. In: Proceedings of the 3rd International Workshop on Automation of Software Test, pp. 67–70. ACM Press, New York (2008)Google Scholar
  19. 19.
    Felderer, M., Ramler, R.: Experiences and challenges of introducing risk-based testing in an industrial project. In: Software Quality. Increasing Value in Software and Systems Development, pp. 10–29. Springer, Berlin (2013)Google Scholar
  20. 20.
    Yoon, H., Choi, B.: A test case prioritization based on degree of risk exposure and its empirical study. Int. J. Softw. Eng. Knowl. Eng. 21(02), 191–209 (2011)CrossRefGoogle Scholar
  21. 21.
    Souza, E., Gusmão, C., Venâncio, J.: Risk-based testing: a case study. In: IEEE 2010 Seventh International Conference on Information Technology: New Generations (ITNG), pp. 1032–1037 (2010)Google Scholar
  22. 22.
    Souza, E., Gusmao, C., Alves, K., Venancio, J., Melo, R.: Measurement and control for risk-based test cases and activities. In: 10th Latin American Test Workshop, pp. 1–6. IEEE Press, New York (2009)Google Scholar
  23. 23.
    Borland: SilkCentral (2013). Available at Accessed November 30 2013
  24. 24.
    Microtool: in-Step (2013). Available at Accessed November 30 2013
  25. 25.
    McCabe, T.: A complexity measure. IEEE Trans. Softw. Eng. 308–320 (1976)Google Scholar
  26. 26.
    SonarSource: Sonar (2013). Available at Accessed November 30 2013
  27. 27.
    Felderer, M., Beer, A.: Using defect taxonomies to improve the maturity of the system test process: results from an industrial case study. In: Software Quality. Increasing Value in Software and Systems Development, LNBIP 133, pp. 125–146. Springer, Berlin (2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.University of InnsbruckInnsbruckAustria
  2. 2.Software Competence Center HagenbergHagenbergAustria

Personalised recommendations