A taxonomy of risk-based testing

Introduction

Abstract

Software testing has often to be done under severe pressure due to limited resources and a challenging time schedule facing the demand to assure the fulfillment of the software requirements. In addition, testing should unveil those software defects that harm the mission-critical functions of the software. Risk-based testing uses risk (re-)assessments to steer all phases of the test process to optimize testing efforts and limit risks of the software-based system. Due to its importance and high practical relevance, several risk-based testing approaches were proposed in academia and industry. This paper presents a taxonomy of risk-based testing providing a framework to understand, categorize, assess, and compare risk-based testing approaches to support their selection and tailoring for specific purposes. The taxonomy is aligned with the consideration of risks in all phases of the test process and consists of the top-level classes risk drivers, risk assessment, and risk-based test process. The taxonomy of risk-based testing has been developed by analyzing the work presented in available publications on risk-based testing. Afterwards, it has been applied to the work on risk-based testing presented in this special section of the International Journal on Software Tools for Technology Transfer.

Keywords

Risk-based testing Risk management Risk analysis Software testing Classification Taxonomy 

References

  1. 1.
    Amland, S.: Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. J. Syst. Softw. 53(3), 287–295 (2000)CrossRefGoogle Scholar
  2. 2.
    Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press, Cambridge (2008)CrossRefMATHGoogle Scholar
  3. 3.
    Bach, J.: Heuristic risk-based testing. Softw. Test. Qual. Eng. Mag. 11, 99 (1999)Google Scholar
  4. 4.
    Bai, X., Kenett, R.S.: Risk-based adaptive group testing of semantic web services. In: 33rd Annual IEEE international computer software and applications conference (COMPSAC’09). vol. 2, pp. 485–490. IEEE (2009)Google Scholar
  5. 5.
    Bai, X., Kenett, R.S., Yu, W.: Risk assessment and adaptive group testing of semantic web services. Int. J. Softw. Eng. Knowl. Eng. 22(05), 595–620 (2012)CrossRefGoogle Scholar
  6. 6.
    Briand, L.C., Labiche, Y., He, S.: Automating regression test selection based on UML designs. Inf. Softw. Technol. 51(1), 16–30 (2009)CrossRefGoogle Scholar
  7. 7.
    Carrozza, G., Pietrantuono, R., Russo, S.: Dynamic test planning: a study into an industrial context. STTT in this volume (2014)Google Scholar
  8. 8.
    Casado, R., Tuya, J., Younas, M.: Testing long-lived web services transactions using a risk-based approach. In: 10th international conference on quality software. pp. 337–340. IEEE (2010)Google Scholar
  9. 9.
    Chen, Y., Probert, R.L., Sims, D.P.: Specification-based regression test selection with risk analysis. In: proceedings of the 2002 conference of the Centre for Advanced Studies on Collaborative research. p. 1. IBM Press (2002)Google Scholar
  10. 10.
    Erdogan, G., Li, Y., Runde, R.K., Seehusen, F., Stølen, K.: Approaches for the combined use of risk analysis and testing: a systematic literature review. STTT in this volume (2014)Google Scholar
  11. 11.
    Felderer, M., Beer, A.: Using defect taxonomies to improve the maturity of the system test process: results from an industrial case study. In: software quality. Increasing value in software and systems development, pp. 125–146. Springer (2013)Google Scholar
  12. 12.
    Felderer, M., Haisjackl, C., Breu, R., Motz, J.: Integrating manual and automatic risk assessment for risk-based testing, pp. 159–180. Software quality. Process automation in software, development (2012)Google Scholar
  13. 13.
    Felderer, M., Haisjackl, C., Pekar, V., Breu, R.: A risk assessment framework for software testing. In: ISoLA 2014. Springer (2014)Google Scholar
  14. 14.
    Felderer, M., Ramler, R.: Experiences and challenges of introducing risk-based testing in an industrial project. In: Software quality. Increasing value in software and systems development, pp. 10–29. Springer (2013)Google Scholar
  15. 15.
    Felderer, M., Ramler, R.: Integrating risk-based testing in industrial test processes. Softw. Qual. J. 22(3), 543–575 (2014)CrossRefGoogle Scholar
  16. 16.
    Felderer, M., Ramler, R.: A multiple case study on risk-based testing in industry. STTT in this volume (2014)Google Scholar
  17. 17.
    Fredriksen, R., Kristiansen, M., Gran, B.A., Stølen, K., Opperud, T.A., Dimitrakos, T.: The coras framework for a model-based risk management process. In: Anderson, S., Bologna, S., Felici, M. (eds.) SAFECOMP. Lecture Notes in Computer Science, vol. 2434, pp. 94–105. Springer (2002)Google Scholar
  18. 18.
    Gerrard, P., Thompson, N.: Risk-based e-business testing. Artech House Publishers, (2002)Google Scholar
  19. 19.
    Goel, A.L.: Software reliability models: assumptions, limitations, and applicability. IEEE Trans. Softw. Eng. 11(12), 1411–1423 (Dec 1985)Google Scholar
  20. 20.
    Graham, D., Fewster, M.: Experiences of test automation: case studies of software test automation. Addison-Wesley Professional, (2012)Google Scholar
  21. 21.
    Hosseingholizadeh, A.: A source-based risk analysis approach for software test optimization. In: Computer Engineering and Technology (ICCET), 2010 2nd international conference on. vol. 2, pp. V2601–V2604. IEEE (2010)Google Scholar
  22. 22.
    Huizinga, D., Kolawa, A.: Automated defect prevention: best practices in software management. Wiley (2007)Google Scholar
  23. 23.
    IEEE: IEEE Standard for Software and System Test Documentation. IEEE Std 829–2008 (2008)Google Scholar
  24. 24.
    ISO: ISO 14971: medical devices—application of risk management to medical devices. ISO (2000)Google Scholar
  25. 25.
    ISO: ISO/IEC/IEEE 29119 Software Testing. http://www.softwaretestingstandard.org/ (2013). Accessed 6 May 2014
  26. 26.
    ISTQB: Standard glossary of terms used in software testing. version 2.2. Tech. rep., ISTQB (2012)Google Scholar
  27. 27.
    Jorgensen, M., Boehm, B., Rifkin, S.: Software development effort estimation: formal models or expert judgment? IEEE Softw. 26(2), 14–19 (2009)CrossRefGoogle Scholar
  28. 28.
    Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Tech. rep., Technical report, EBSE Technical Report EBSE-2007-01 (2007)Google Scholar
  29. 29.
    Kloos, J., Hussain, T., Eschbach, R.: Risk-based testing of safety-critical embedded systems driven by fault tree analysis. In: ICSTW 2011. pp. 26–33. IEEE (2011)Google Scholar
  30. 30.
    Kumar, N., Sosale, D., Konuganti, S.N., Rathi, A.: Enabling the adoption of aspects-testing aspects: a risk model, fault model and patterns. In: proceedings of the 8th ACM international conference on Aspect-oriented software development. pp. 197–206. ACM (2009)Google Scholar
  31. 31.
    Murthy, K.K., Thakkar, K.R., Laxminarayan, S.: Leveraging risk based testing in enterprise systems security validation. In: first international conference on emerging network intelligence. pp. 111–116. IEEE (2009)Google Scholar
  32. 32.
    Neubauer, J., Windmüller, S., Steffen, B.: Risk-based testing via active continuous quality control. STTT in this volume (2014)Google Scholar
  33. 33.
    Radatz, J., Geraci, A., Katki, F.: IEEE standard glossary of software engineering terminology. IEEE Std. 610121990, 121990 (1990)Google Scholar
  34. 34.
    Ray, M., Mohapatra, D.P.: Risk analysis: a guiding force in the improvement of testing. IET Softw. 7(1), 29–46 (2013)CrossRefGoogle Scholar
  35. 35.
    Redmill, F.: Exploring risk-based testing and its implications. Softw. Test. Verif. Reliab. 14(1), 3–15 (2004)CrossRefGoogle Scholar
  36. 36.
    Redmill, F.: Theory and practice of risk-based testing. Softw. Test. Verif. Reliab. 15(1), 3–20 (2005)CrossRefGoogle Scholar
  37. 37.
    Rosenberg, L., Stapko, R., Gallo, A.: Risk-based object oriented testing. Proceedings of 13th international software/internet quality week-QW 2 (2000)Google Scholar
  38. 38.
    Schieferdecker, I., Grossmann, J., Schneider, M.: Model-based security testing. Proceedings 7th workshop on model-based testing (2012)Google Scholar
  39. 39.
    Souza, E., Gusmao, C., Alves, K., Venancio, J., Melo, R.: Measurement and control for risk-based test cases and activities. In: 10th Latin American test workshop. pp. 1–6. IEEE (2009)Google Scholar
  40. 40.
    Souza, E., Gusmão, C., Venâncio, J.: Risk-based testing: A case study. In: information technology: new generations (ITNG), 2010 seventh international conference on. pp. 1032–1037. IEEE (2010)Google Scholar
  41. 41.
    Stallbaum, H., Metzger, A.: Employing requirements metrics for automating early risk assessment. Proceedings of MeReP07, Palma de Mallorca, Spain. pp. 1–12 (2007)Google Scholar
  42. 42.
    Stallbaum, H., Metzger, A., Pohl, K.: An automated technique for risk-based test case generation and prioritization. In: Proceedings of the 3rd international workshop on Automation of software test. pp. 67–70. ACM (2008)Google Scholar
  43. 43.
    Stallbaum, H., Metzger, A., Pohl, K.: An automated technique for risk-based test case generation and prioritization. In: proceedings of the 3rd international workshop on automation of software test. pp. 67–70. AST ’08, ACM, New York, NY, USA (2008)Google Scholar
  44. 44.
    Standards Australia/New Zealand: risk management AS/NZS 4360:2004 (2004)Google Scholar
  45. 45.
    Tran, V., Liu, D.B.: A risk-mitigating model for the development of reliable and maintainable large-scale commercial-off-the-shelf integrated software systems. In: reliability and maintainability symposium. 1997 proceedings, annual. pp. 361–367 (1997)Google Scholar
  46. 46.
    van Veenendaal, E.: Practical risk-based testing—The PRISMA Approach. UTN Publishers (2012)Google Scholar
  47. 47.
    Wendland, M.F., Kranz, M., Schieferdecker, I.: A systematic approach to risk-based testing using risk-annotated requirements models. ICSEA 2012, 636–642 (2012)Google Scholar
  48. 48.
    Windmüller, S., Neubauer, J., Steffen, B., Howar, F., Bauer, O.: Active continuous quality control. In: proceedings of the 16th international ACM sigsoft symposium on component-based software engineering. pp. 111–120. ACM (2013)Google Scholar
  49. 49.
    Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. 22(2), 67–120 (Mar 2012) Google Scholar
  50. 50.
    Yoon, H., Choi, B.: A test case prioritization based on degree of risk exposure and its empirical study. Int. J. Softw. Eng. Know. Eng. 21(02), 191–209 (2011)CrossRefGoogle Scholar
  51. 51.
    Zech, P.: Risk-based security testing in cloud computing environments. In: ICST 2011. pp. 411–414. IEEE (2011)Google Scholar
  52. 52.
    Zech, P., Felderer, M., Breu, R.: Towards risk-driven security testing of service centric systems. In: QSIC. pp. 140–143 (2012)Google Scholar
  53. 53.
    Zimmermann, F., Eschbach, R., Kloos, J., Bauer, T., et al.: Risk-based statistical testing: A refinement-based approach to the reliability analysis of safety-critical systems. In: EWDC 2009 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.University of InnsbruckInnsbruckAustria
  2. 2.Fraunhofer Institute FOKUS and Freie Universität BerlinBerlinGermany

Personalised recommendations