Coverage-Based Testing with Symbolic Transition Systems
We provide a model-based testing approach for systems comprising both state-transition based control flow, and data elements such as variables and data-dependent transitions. We propose test generation and execution, based on model-coverage: we generate test cases that aim to reach all transitions of the model. To obtain a test case reaching a certain transition, we need to combine reachability in the control flow, and satisfiability of the data elements of the model. Concrete values for data parameters are generated on-the-fly, i.e., during test execution, such that received outputs from the system can be taken into account for the inputs later provided in test execution. Due to undecidability of the satisfiability problem, SMT solvers may return result ‘unknown’. Our algorithm deals with this explicitly. We implemented our method in Maude combined with Z3, and use this to demonstrate the applicability of our method on the Bounded Retransmission Protocol benchmark. We measure performance by counting the number of inputs and outputs needed to discover bugs in mutants, i.e., in non-conforming variants of the specification. As a result, we find that we perform 3 times better, according to the geometric mean, than when using random testing as implemented by the tool TorXakis.
This work was largely performed while the first author was visiting INRIA/the University of Lille. We would like to thank Vlad Rusu for making this visit possible, and for his feedback and support with using Maude.
- 1.Automata Wiki. automata.cs.ru.nl
- 2.Homepage Petra van den Bos. petravdbos.nl
- 4.van den Bos, P., Stoelinga, M.: Tester versus bug: a generic framework for model-based testing via games. In: Orlandini, A., Zimmermann, M. (eds.) Proceedings Ninth International Symposium on Games, Automata, Logics, and Formal Verification. Electronic Proceedings in Theoretical Computer Science, 26–28th September 2018, Saarbrücken, Germany, vol. 277, pp. 118–132. Open Publishing Association (2018)Google Scholar
- 5.Clavel, M., et al.: Maude: specification and programming in rewriting logic. Theoret. Comput. Sci. 285(2), 187–243 (2002). Rewriting Logic and its ApplicationsGoogle Scholar
- 15.Huijben, M.: Efficient constrained random sampling for use in a model based testing tool. Master’s thesis, Institute for Computing and Information Sciences, Radboud University, Nijmegen, The Netherlands (2019)Google Scholar
- 16.Iyer, M.A.: Race: a word-level ATPG-based constraints solver system for smart random simulation, p. 299. Citeseer (2003)Google Scholar
- 19.Kitchen, N.: Markov Chain Monte Carlo stimulus generation for constrained random simulation. Ph.D. thesis, UC Berkeley (2010)Google Scholar
- 20.Li, J.J., Wong, W.E.: Automatic test generation from communicating extended finite state machine (CEFSM)-based models. In: Proceedings Fifth IEEE International Symposium on Object-Oriented Real-Time Distributed Computing, ISIRC 2002, pp. 181–185, April 2002Google Scholar
- 21.Petrenko, A.: Checking experiments for symbolic input/output finite state machines. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 229–237, April 2016Google Scholar
- 22.TorXakis. https://github.com/torxakis