Model-Based Mutation Testing of an Industrial Measurement Device

  • Bernhard K. Aichernig
  • Jakob Auer
  • Elisabeth Jöbstl
  • Robert Korošec
  • Willibald Krenn
  • Rupert Schlick
  • Birgit Vera Schmidt
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8570)


MoMuT::UML is a model-based mutation testing tool for UML models. It maps UML state machines to a formal semantics and performs a conformance check between an original and a set of mutated models to automatically generate test cases. The resulting test suite is able to detect whether a system under test implements one of the faulty models instead of the correct, original model. In this work, we illustrate the whole model-based mutation testing process by means of an industrial case study. We test the control logic of a device that counts the particles in exhaust gases. First, we model the system under test in UML. Then, MoMuT::UML is used to automatically generate three test suites from the UML test model: one mutation-based test suite, one set of random test cases, and a third test suite combining random and mutation-based test case generation. The test cases are executed on the system under test and effectively reveal several errors. Finally, we compare the fault detection capabilities of the three test suites on a set of faulty systems, which were created by intentionally injecting faults into the implementation.


test case generation model-based testing mutation testing automotive industry UML 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aggarwal, M., Sabharwal, S.: Test case generation from UML state machine diagram: A survey. In: ICCCT, pp. 133–140. IEEE (2012)Google Scholar
  2. 2.
    Aichernig, B.K., Brandl, H., Jöbstl, E., Krenn, W., Schlick, R., Tiran, S.: Killing strategies for model-based mutation testing. Software Testing, Verification and Reliability (2014)Google Scholar
  3. 3.
    Aichernig, B.K., Jöbstl, E.: Efficient refinement checking for model-based mutation testing. In: QSIC, pp. 21–30. IEEE (2012)Google Scholar
  4. 4.
    Aichernig, B.K., Jöbstl, E.: Towards symbolic model-based mutation testing: Combining reachability and refinement checking. MBT. EPTCS 80, 88–102 (2012)CrossRefGoogle Scholar
  5. 5.
    Aichernig, B.K., Jöbstl, E., Kegele, M.: Incremental refinement checking for test case generation. In: Veanes, M., Viganò, L. (eds.) TAP 2013. LNCS, vol. 7942, pp. 1–19. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  6. 6.
    Aichernig, B.K., Peischl, B., Weiglhofer, M., Wotawa, F.: Protocol conformance testing a SIP registrar: An industrial application of formal methods. In: SEFM, pp. 215–224. IEEE (2007)Google Scholar
  7. 7.
    Ammann, P., Black, P.E., Majurski, W.: Using model checking to generate tests from specifications. In: ICFEM, pp. 46–54. IEEE (1998)Google Scholar
  8. 8.
    Auer, J.: Automated Integration Testing of Measurement Devices - A Case Study at AVL List GmbH. Bachelor’s thesis, Graz University of Technology (2013)Google Scholar
  9. 9.
    Back, R.J., Kurki-Suonio, R.: Decentralization of process nets with centralized control. In: PODC, pp. 131–142. ACM (1983)Google Scholar
  10. 10.
    Boroday, S., Petrenko, A., Groz, R.: Can a model checker generate tests for non-deterministic systems? ENCTS 190(2), 3–19 (2007)Google Scholar
  11. 11.
    Brandl, H., Weiglhofer, M., Aichernig, B.K.: Automated conformance verification of hybrid systems. In: QSIC, pp. 3–12. IEEE (2010)Google Scholar
  12. 12.
    Briand, L.C., Labiche, Y., Cui, J.: Automated support for deriving test requirements from UML statecharts. Software and System Modeling 4(4), 399–423 (2005)CrossRefGoogle Scholar
  13. 13.
    Budd, T., Gopal, A.: Program testing by specification mutation. Comput. Lang. 10(1), 63–73 (1985)CrossRefMATHGoogle Scholar
  14. 14.
    DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: Help for the practicing programmer. IEEE Computer 11(4), 34–41 (1978)CrossRefGoogle Scholar
  15. 15.
    Fabbri, S.C.P.F., Maldonado, J.C., Masiero, P.C., Delamaro, M.E., Wong, W.E.: Mutation testing applied to validate specifications based on Statecharts. In: ISSRE, pp. 210–219. IEEE (1999)Google Scholar
  16. 16.
    Fröhlich, P., Link, J.: Automated test case generation from dynamic models. In: Bertino, E. (ed.) ECOOP 2000. LNCS, vol. 1850, pp. 472–492. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  17. 17.
    Gnesi, S., Latella, D., Massink, M.: Formal test-case generation for UML statecharts. In: ICECCS, pp. 75–84. IEEE (2004)Google Scholar
  18. 18.
    Grub, P., Takang, A.A.: Software Maintenance: Concepts and Practice, 2nd edn. World Scientific Publishing (2003)Google Scholar
  19. 19.
    Hoare, C., He, J.: Unifying Theories of Programming. Prentice Hall (1998)Google Scholar
  20. 20.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Software Eng. 37(5), 649–678 (2011)CrossRefGoogle Scholar
  21. 21.
    Jöbstl, E.: Model-Based Mutation Testing with Constraint and SMT Solvers. Ph.D. thesis, Graz University of Technology, Institute for Software Technology (2014)Google Scholar
  22. 22.
    Kansomkeat, S., Rivepiboon, W.: Automated-generating test case using UML statechart diagrams. In: SAICSIT, pp. 296–300 (2003)Google Scholar
  23. 23.
    Krenn, W., Schlick, R., Aichernig, B.K.: Mapping UML to labeled transition systems for test-case generation - a translation via object-oriented action systems. In: de Boer, F.S., Bonsangue, M.M., Hallerstede, S., Leuschel, M. (eds.) FMCO 2009. LNCS, vol. 6286, pp. 186–207. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  24. 24.
    Myers, G.J.: The Art of Software Testing. Wiley (1979)Google Scholar
  25. 25.
    Nelson, G.: A generalization of Dijkstra’s calculus. ACM Trans. Program. Lang. Syst. 11(4), 517–561 (1989)CrossRefGoogle Scholar
  26. 26.
    Nogueira, S., Sampaio, A., Mota, A.: Guided test generation from CSP models. In: Fitzgerald, J.S., Haxthausen, A.E., Yenigun, H. (eds.) ICTAC 2008. LNCS, vol. 5160, pp. 258–273. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  27. 27.
    Offutt, J., Abdurazik, A.: Generating tests from UML specifications. In: France, R.B. (ed.) UML 1999. LNCS, vol. 1723, pp. 416–429. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  28. 28.
    Okun, V., Black, P.E., Yesha, Y.: Testing with model checker: Insuring fault visibility. In: 2002 WSEAS Int. Conf. on System Science, Applied Mathematics & Computer Science, and Power Engineering Systems, pp. 1351–1356 (2003)Google Scholar
  29. 29.
    Paradkar, A.: Case studies on fault detection effectiveness of model based test generation techniques. SIGSOFT Softw. Eng. Notes 30(4), 1–7 (2005)CrossRefGoogle Scholar
  30. 30.
    Seifert, D.: Conformance testing based on UML state machines. In: Liu, S., Araki, K. (eds.) ICFEM 2008. LNCS, vol. 5256, pp. 45–65. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  31. 31.
    Stocks, P.A.: Applying formal methods to software testing. Ph.D. thesis, Department of computer science, University of Queensland (1993)Google Scholar
  32. 32.
    Tretmans, J.: Test generation with inputs, outputs and repetitive quiescence. Software - Concepts and Tools 17(3), 103–120 (1996)MATHGoogle Scholar
  33. 33.
    Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers (2007)Google Scholar
  34. 34.
    Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Software Testing, Verification and Reliability 22(5), 297–312 (2012)CrossRefGoogle Scholar
  35. 35.
    Weissenbacher, G. (ed.): D 3.2b - modelling languages (final version). Tech. rep., MOGENTES (2010)Google Scholar
  36. 36.
    Weißleder, S.: Test Models and Coverage Criteria for Automatic Model-Based Test Generation with UML State Machines. Ph.D. thesis, Humboldt Universität zu Berlin (2009)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Bernhard K. Aichernig
    • 1
  • Jakob Auer
    • 2
  • Elisabeth Jöbstl
    • 1
  • Robert Korošec
    • 2
  • Willibald Krenn
    • 3
  • Rupert Schlick
    • 3
  • Birgit Vera Schmidt
    • 2
  1. 1.Institute for Software TechnologyGraz University of TechnologyAustria
  2. 2.AVL List GmbHGrazAustria
  3. 3.AIT Austrian Institute of TechnologyViennaAustria

Personalised recommendations