Abstract
Risk orientation in testing is an important means to balance quality, time-to-market, and cost of software. Especially for small and medium enterprises (SME) under high competitive and economic pressure, risk orientation can help to focus testing activities on critical areas of a software product. Although several risk-based approaches to testing are available, the topic has so far not been investigated in the context of SME, where risks are often associated with business critical issues. This article fills the gap and explores the state of risk orientation in the testing processes of SME. Furthermore, it compares the state of risk-based testing in SME to the situation in large enterprises. The article is based on a multiple case study conducted with five SME. A previous study on risk-based testing in large enterprises is used as reference for investigating the differences between risk orientation in SME and large enterprises. The findings of our study show that a strong business focus, the use of informal risk concepts, as well as the application of risk knowledge to reduce testing cost and time are key differences of risk-based testing in SME compared to large enterprises.
Similar content being viewed by others
References
Ahern, D., Clouse, A., & Turner, R. (2008). CMMI distilled: A practical introduction to integrated process improvement. Boston: Addison-Wesley Professional.
Amland, S. (2000). Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. Journal of Systems and Software, 53(3), 287–295.
Azar, J., Smith, R. K., & Cordes, D. (2007). Value-oriented requirements prioritization in a small development organization. IEEE Software, 24(1), 32–37.
Bath, G., & van Veenendaal, E. (2014). Improving the test process. Rocky Nook: Massachusetts.
Dorling, A. (1993). SPICE: Software process improvement and capability determination. Software Quality Journal, 2(4), 209–224.
Erdogan, G., Li, Y., Runde, R. K., Seehusen, F., & Stølen, K. (2014). Approaches for the combined use of risk analysis and testing: A systematic literature review. International Journal on Software Tools for Technology Transfer, 16(5), 627–642.
Felderer, M., Haisjackl, C., Breu, R., & Motz, J. (2012). Integrating manual and automatic risk assessment for risk-based testing. Software quality. In Process automation in software development. 4th international conference SWQD (pp. 159–180).
Felderer, M., & Ramler, R. (2013). Experiences and challenges of introducing risk-based testing in an industrial project. In Software quality. Increasing value in software and systems development. 5th international conference SWQD (pp. 10–29).
Felderer, M., & Ramler, R. (2014a). Integrating risk-based testing in industrial test processes. Software Quality Journal, 22(3), 543–575.
Felderer, M., & Ramler, R. (2014b). A multiple case study on risk-based testing in industry. International Journal on Software Tools for Technology Transfer, 16(5), 609–625.
Felderer, M., & Schieferdecker, I. (2014). A taxonomy of risk-based testing. International Journal on Software Tools for Technology Transfer, 16(5), 559–568.
Felderer, M., Wendland, M-F., & Schieferdecker, I. (2014). Risk-based testing. In Leveraging applications of formal methods, verification and validation. Specialized techniques and applications (pp. 274–276). Berlin, Heidelberg: Springer.
García, I., Pacheco, C., Mendoza, E., Calvo Manzano, J. A., Cuevas, G., & San Feliu, T. (2012). Managing the software process with a software process improvement tool in a small enterprise. Journal of Software: Evolution and Process, 24(5), 481–491.
Gerrard, P., & Thompson, N. (2002). Risk based e-business testing. Norwood: Artech House Inc.
Gleirscher, M., Golubitskiy, D., Irlbeck, M., & Wagner, S. (2014). Introduction of static quality analysis in small-and medium-sized software enterprises: Experiences from technology transfer. Software Quality Journal, 22(3), 499–542.
Harrold, M. J. (2000). Testing: A roadmap. In Proceedings of the conference on the future of software engineering (pp. 61–72).
ISO/IEC. (2011). Software engineering—Lifecycle profiles for Very Small Entities (VSEs). Available online at http://www.iso.org/iso/catalogue_detail?csnumber=51150. Accessed on July 15, 2015.
ISO/IEC/IEEE. (2013). ISO/IEC/IEEE 29119 Software testing. Draft available online at http://www.softwaretestingstandard.org/. Accessed on July 15, 2015.
ISTQB. (2012). Standard glossary of terms used in software testing. Version 2.2. Brussels: International Software Testing Qualifications Board.
Karlström, D., Runeson, P., & Norden, S. (2005). A minimal test practice framework for emerging software organizations. Software Testing, Verification and Reliability, 15(3), 145–166.
Karolak, D. W. (1995). Software engineering risk management. Hoboken: Wiley-IEEE Computer Society Press.
Kautz, K. (1999). Making sense of measurement for small organizations. IEEE Software, 16, 14–20.
Koomen, T., & Pol, M. (1999). Test process improvement: A practical step-by-step guide to structured testing. Boston: Addison-Wesley Professional.
Koomen, T., van der Aalst, L., Broekman, B., & Vroon, M. (2006). TMap next, for result-driven testing. ‘s-Hertogenbosch: UTN Publishers.
Martin, K., & Hoffman, B. (2007). An open source approach to developing software in a small organization. IEEE Software, 24(1), 46–53.
Martin, D., Rooksby, J., Rouncefield, M., & Sommerville, I. (2007). ‘Good’ organisational reasons for ‘Bad’ software testing: An ethnographic study of testing in a small software company. In 29th international conference on software engineering (ICSE ‘07).
Mc Caffery, F., Taylor, P. S., & Coleman, G. (2007). Adept: A unified assessment method for small software companies. IEEE Software, 24(1), 24–31.
Mishra, D., & Mishra, A. (2008). Software process improvement methodologies for small and medium enterprises. In 9th international conference on product-focused software process improvement (PROFES 2008).
Pino, F. J., García, F., & Piattini, M. (2008). Software process improvement in small and medium software enterprises: A systematic review. Software Quality Journal, 16(2), 237–261.
Pino, F. J., Pardo, C., García, F., & Piattini, M. (2010). Assessment methodology for software process improvement in small organizations. Information and Software Technology, 52(10), 1044–1061.
Redmill, F. (2004). Exploring risk-based testing and its implications. Software Testing Verification and Reliability, 14(1), 3–15.
Redmill, F. (2005). Theory and practice of risk-based testing: Research Articles. Software Testing Verification and Reliability, 15(1), 3–20.
Richardson, I., & von Wangenheim, C. G. (2007). Why are small software organizations different. IEEE Software, 24(1), 18–22.
Runeson, P., Höst, M., Rainer, A., & Regnell, B. (2012). Case study research in software engineering: Guidelines and examples. Hoboken: Wiley.
Sanchez-Gordon, M. L., O’Connor, R. V., & Colomo-Palacios, R. (2015). Evaluating VSEs viewpoint and sentiment towards the ISO/IEC 29110 standard: A two country grounded theory study. In 15th international conference on software process improvement and capability determination (SPICE 2015).
Souza, E., Gusmão, C., & Venâncio, J. (2010). Risk-based testing: A case study. In seventh international conference on information technology: New generations (ITNG) (pp. 1032–1037).
Souza, E., Gusmão, C., Venâncio, J., & Melo, R. (2009). Measurement and control for risk-based test cases and activities. In 10th Latin American Test Workshop (LATW’09) (pp. 1–6).
Steiner, M., Blaschke, M., Philipp, M., & Schweigert, T. (2012). Make test process assessment similar to software process assessment—The test SPICE approach. Journal of Software: Evolution and Process, 24(5), 471–480.
van Veenendaal, E., Goslin, A., Olsen, K., O’Hara, F., Miller, M., Thompson, G., & Wells, B. (2008). Test Maturity Model integration (TMMi) Version 1.0.. TMMi Foundation: Princeton.
Yin, R. K. (2014). Case study research: Design and methods. Thousand Oaks: Sage Publications.
Yoon, H., & Choi, B. (2011). A test case prioritization based on degree of risk exposure and its empirical evaluation. International Journal of Software Engineering and Knowledge Engineering, 21(02), 191–209.
Acknowledgments
This work has been supported by the research project Smart Testing funded by the Austrian Research Promotion Agency (FFG), the COMET Competence Center program of the Austrian Research Promotion Agency (FFG), and the project QE LaB—Living Models for Open Systems (www.qe-lab.at) funded by the Austrian Federal Ministry of Economics (Bundesministerium für Wirtschaft und Arbeit).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Felderer, M., Ramler, R. Risk orientation in software testing processes of small and medium enterprises: an exploratory and comparative study. Software Qual J 24, 519–548 (2016). https://doi.org/10.1007/s11219-015-9289-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-015-9289-z