Risk-based testing via active continuous quality control

  • Johannes NeubauerEmail author
  • Stephan Windmüller
  • Bernhard Steffen


In this paper, we show how our approach to active continuous quality control (ACQC), which employs learning technology to automatically maintain test models along the whole life cycle, can be extended to include risk analysts for supporting risk-based testing. Key to this enhancement is the tailoring of ACQC’s characteristic automata learning-based model extraction to prioritize critical aspects. Technically, risk analysts are provided with an abstract modeling level tailored to design test components (learning symbols) that encompass data-flow constraints reflecting a given risk profile. The resulting alphabet models are already sufficient to steer the ACQC process in a fashion that increases the risk coverage, while it at the same time radically reduces the testing effort. We illustrate our approach by means of case studies with Springer’s Online Conference Service (OCS) which show the impact of the risk prioritization on the performance: risk-based regression testing tailored for system migration and for pure functional evolution.


Risk-based testing Active automata learning Testing Process modeling 


  1. 1.
    Aarts, F., Jonsson, F., Uijen, J.: Generating models of infinite-state communication protocols using regular inference with abstraction. In: ICTSS ’10, Volume 6435 of LNCS, pp. 188–204. Springer, Berlin (2010)Google Scholar
  2. 2.
    Aarts, F., Schmaltz, J., Vaandrager, F.: Inference and abstraction of the biometric passport. In: ISoLA 2010, Volume 6415 of LNCS, pp. 673–686. Springer, Berlin (2010)Google Scholar
  3. 3.
    Angluin, D.: Learning regular sets from queries and counterexamples. Inf. Comput. 75(2), 87–106 (1987)CrossRefzbMATHMathSciNetGoogle Scholar
  4. 4.
    Bach, J.: Heuristic risk-based testing. Softw. Test. Quality Eng. Mag. 11, 99 (1999)Google Scholar
  5. 5.
    Bauer, O., Neubauer, J., Steffen, B., Howar, F.: Reusing system states by active learning algorithms. In: Moschitti, A., Scandariato, R. (eds.) Eternal Systems, Volume 255 of CCSE. Springer, Berlin (2012)Google Scholar
  6. 6.
    Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In: FASE ’05, Volume 3442 of LNCS, pp. 175–189. Springer, Berlin (2005)Google Scholar
  7. 7.
    Beydeda, S., Gruhn, V.: Integrating white- and black-box techniques for class-level regression testing. In: COMPSAC ’01, Washington, DC, USA, pp. 357–362. IEEE Computer Society (2001)Google Scholar
  8. 8.
    Bossert, V., Hiet, G., Henin, G.: Modelling to simulate botnet command and control protocols for the evaluation of network intrusion detection systems. In: SAR-SSI ’11, pp. 1–8. IEEE Computer Society (2011)Google Scholar
  9. 9.
    Chen, Y., Probert, R.L., Ural, H.: Model-based regression test suite generation using dependence analysis. In: A-MOST ’07, New York, NY, USA, pp. 54–62. ACM (2007)Google Scholar
  10. 10.
    Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. 4(3), 178–187 (1978)CrossRefzbMATHGoogle Scholar
  11. 11.
    Felderer, M., Haisjackl, M., Breu, R., Motz, J.: Integrating manual and automatic risk assessment for risk-based testing. In: Biffl, S., Winkler, D., Bergsmann, J. (eds.) Software Quality, Process Automation in Software Development, Volume 94 of Lecture Notes in Business Information Processing, pp. 159–180. Springer, Berlin (2012)CrossRefGoogle Scholar
  12. 12.
    Felderer, M., Ramler, R.: A multiple case study on risk-based testing in industry. sttt-RBT, Under, Review (2014)Google Scholar
  13. 13.
    Fowler, M.: Inversion of control containers and the dependency injection pattern (2004) .
  14. 14.
    Gerrard, P., Thompson, N.: Risk based e-business resting. Artech House (2002)Google Scholar
  15. 15.
    Gnesi, S., Margaria, T.: Formal Methods for Industrial Critical Systems: A Survey of Applications. Wiley, :London (2012)CrossRefGoogle Scholar
  16. 16.
    Grabowski, J., Hogrefe, D., Réthy, G., Schieferdecker, I., Wiles, A., Willcock, C.: An introduction to the testing and test control notation (ttcn-3). Comput. Netw. 42(3), 375–403 (2003)CrossRefzbMATHGoogle Scholar
  17. 17.
    Grabowski, J., Wiles, A., Willcock, C., Hogrefe, D.: On the design of the new testing language ttcn-3. In: Ural, H., Probert, R., Bochmann, G. (eds.) Testing of Communicating Systems, Volume 48 of IFIP Advances in Information and Communication Technology, pp. 161–176. Springer, New York (2000)Google Scholar
  18. 18.
    Grossmann, J.: Testing hybrid systems with ttcn-3 embedded. sttt-RBT. Under Review (2014)Google Scholar
  19. 19.
    Hagerer, A. Hungar, H., Niese, O., Steffen, B.: Model generation by moderated regular extrapolation. In FASE ’02, pp. 80–95, London, UK. Springer, Berlin (2002)Google Scholar
  20. 20.
    Howar, F., Steffen, B., Jonsson, B., Cassel, S.: Inferring canonical register automata. In: Kuncak, V., Rybalchenko, A. (eds.) Verification, Model Checking, and Abstract Interpretation, Volume 7148 of Lecture Notes in Computer Science, pp. 251–266. Springer, Berlin (2012)Google Scholar
  21. 21.
    Howar, F., Steffen, B., Merten, M.: Automata learning with automated alphabet abstraction refinement. In: Verification, Model Checking, and Abstract Interpretation, Volume 6538 of LNCS, pp. 263–277. Springer, Berlin (2011)Google Scholar
  22. 22.
    Hungar, H., Niese, O., Steffen, B.: Domain-specific optimization in automata learning. In: Computer Aided Verification, Volume 2725 of LNCS, pp. 315–327. Springer, Berlin (2003)Google Scholar
  23. 23.
    Jörges, S., Margaria, T., Steffen, B.: Genesys: service-oriented construction of property conform code generators. ISSE 4(4), 361–384 (2008)Google Scholar
  24. 24.
    Jéron, T.: Symbolic model-based test selection. Electron. Notes Theor. Comput. Sci. 240(0):167–184 (2009). Proceedings of the Eleventh Brazilian Symposium on Formal Methods (SBMF 2008)Google Scholar
  25. 25.
    Jörges, S.: Construction and Evolution of Code Generators: A Model-Driven and Service-Oriented Approach. Springer, Berlin (2013)CrossRefGoogle Scholar
  26. 26.
    Karusseit, M., Margaria, T.: Feature-based modelling of a complex, online-reconfigurable decision support service. In WWV ’05, March 2005. ENTCS 1132Google Scholar
  27. 27.
    Korel, B., Al-Yami, A.M.: Automated regression test generation. SIGSOFT Softw. Eng. Notes 23(2), 143–152 (1998)CrossRefGoogle Scholar
  28. 28.
    Lund, M.S., Solhaug, B.: Model-Driven Risk Analysis. Springer, Berlin (2011)CrossRefGoogle Scholar
  29. 29.
    Marchand, H., Dubreil, J., Jéron, T.: Automatic testing of access control for security properties. In: Núñez, M., Baker, P., Merayo, M. (eds.) Testing of Software and Communication Systems, Volume 5826 of Lecture Notes in Computer Science, pp. 113–128. Springer, Berlin (2009)Google Scholar
  30. 30.
    Margaria, T., Niese, O., Raffelt, H., Steffen. B.: Efficient test-based model generation for legacy reactive systems. In: HLDVT ’04, Washington, DC, USA, pp. 95–100. IEEE Computer Society (2004)Google Scholar
  31. 31.
    Margaria, T., Steffen, B.: Lightweight coarse-grained coordination: a scalable system-level approach. STTT 5(2–3), 107–123 (2004)Google Scholar
  32. 32.
    Margaria, T., Steffen, B.: Agile IT: thinking in user-centric models. In: ISoLA 2009, Volume 17 of Communications in Computer and Information Science, pp. 490–502. Springer, Berlin (2009)Google Scholar
  33. 33.
    Margaria, T., Steffen, B.: Service-orientation: conquering complexity with XMDD. In: Hinchey, M., Coyle, L. (eds.) Conquering Complexity, pp. 217–236. Springer, London (2012)CrossRefGoogle Scholar
  34. 34.
    Merten, M., Steffen, B., Howar, F., Margaria T.: Next generation learnlib. In: TACAS 2011, Volume 6605 of LNCS, pp. 220–223. Springer, Berlin (2011)Google Scholar
  35. 35.
    Müller-Olm, M., Schmidt, D., Steffen, B.: Model-checking: a tutorial introduction. In: Cortesi, A., Filé, G. (eds.) Static Analysis, Volume 1694 of LNCS, pp. 330–354. Springer, Berlin (1999)Google Scholar
  36. 36.
    Neubauer, J., Steffen, B.: Plug-and-play higher-order process integration. Computer 46(11), 56–62 (2013)CrossRefGoogle Scholar
  37. 37.
    Neubauer, J., Steffen, B.: Learning-based cross-platform conformance testing . STVR. Wiley (2014)Google Scholar
  38. 38.
    Neubauer, J. Steffen, B., Margaria, T.: Higher-order process modeling: product-lining, variability modeling and beyond. arXiv, preprint arXiv:1309.5143 (2013)
  39. 39.
    Pickin, S., Jard, C., Jéron, T., Jézéquel, J.-M., Traon, Y.L.: Test synthesis from uml models of distributed software. IEEE Trans. Softw. Eng. 33(4), 252–269 (2007)CrossRefGoogle Scholar
  40. 40.
    Raffelt, H., Merten, M., Steffen, B., Margaria, T.: Dynamic testing via automata learning. STTT 11(4), 307–324 (2009)CrossRefGoogle Scholar
  41. 41.
    Raffelt, H., Steffen, B., Berg, T., Margaria, T.: LearnLib: a framework for extrapolating behavioral models. Int. J. Softw. Tools Technol. Transf. 11(5), 393–407 (2009)CrossRefGoogle Scholar
  42. 42.
    Rivest, R.L., Schapire, R.E.: Inference of finite automata using homing sequences. Inf. Comput. 103(2), 299–347 (1993)CrossRefzbMATHMathSciNetGoogle Scholar
  43. 43.
    Rosenberg, L., Stapko, R., Gallo. A.: Risk-based object oriented testing. In: Proceedings of 13th International Software/Internet Quality Week-QW, San Francisco, California, USA, Vol. 2 (2000)Google Scholar
  44. 44.
    Rothermel, G., Harrold, M.J., Dedhia, J.: Regression test selection for C++ software. Softw. Test. Verif. Reliab. 10, 2000 (1999)Google Scholar
  45. 45.
    Shahbaz, M., Shashidhar, K.C., Eschbach. R.: Iterative refinement of specification for component based embedded systems. In: ISSTA ’11, pp. 276–286. ACM (2011) Google Scholar
  46. 46.
    Smolka, S., Steffen, B.: Priority as extremal probability. Formal Asp. Comput. 8(5), 585–606 (1996)Google Scholar
  47. 47.
    Steffen, B.: Characteristic formulae. In: ICALP, Volume 372 of LNCS, pp. 723–732 (1989)Google Scholar
  48. 48.
    Steffen, B., Howar, F., Merten. M.: Introduction to active automata learning from a practical perspective. In: Bernardo, M., Issarny, V. (eds.) Formal Methods for Eternal Networked Software Systems, Volume 6659 of LNCS, pp. 256–296. Springer, Berlin (2011)Google Scholar
  49. 49.
    Steffen, B., Margaria, T., Nagel, R., Jörges, S., Kubczak, C.: Model-driven development with the jABC. In: Bin, E., Ziv, A., Ur, S. (eds.) Hardware and Software. Verification and Testing, Volume 4383 of Lecture Notes in Computer Science, pp. 92–108. Springer, Berlin (2007)Google Scholar
  50. 50.
    Stepien, B., Peyton, L.: Innovation and evolution in integrated web application testing with ttcn-3. sttt-RBT. Under Review (2014)Google Scholar
  51. 51.
    Tretmans, J.: Model based testing with labelled transition systems. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing, Volume 4949 of LNCS, pp. 1–38. Springer, Berlin (2008)Google Scholar
  52. 52.
    Tretmans, J.: Model-based testing and some steps towards test-based modelling. In: Bernardo, M., Issarny, V. (eds.) Formal Methods for Eternal Networked Software Systems, Volume 6659 of LNCS, pp. 297–326. Springer, Berlin (2011)CrossRefGoogle Scholar
  53. 53.
    Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22(5), 297–312 (2012)CrossRefGoogle Scholar
  54. 54.
    Windmüller, S., Neubauer, J., Steffen, B., Howar, F., Bauer, O.: Active continuous quality control. In: Proceedings of the 16th International ACM Sigsoft symposium on Component-Based Software Engineering, CBSE ’13, New York, NY, USA, pp. 111–120. ACM (2013)Google Scholar
  55. 55.
    Xu, L., Dias, M., Richardson, D.: Generating regression tests via model checking. In: COMPSAC ’04, Washington, DC, USA, pp. 336–341. IEEE Computer Society (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Johannes Neubauer
    • 1
    Email author
  • Stephan Windmüller
    • 1
  • Bernhard Steffen
    • 1
  1. 1.TU Dortmund UniversityDortmundGermany

Personalised recommendations