State machine properties
PBT can also be applied to models in the form of extended finite state machines (EFSMs) [17].
Definition 3
(EFSM) An EFSM can formally be defined as a 6-tuple \( (S, s_0, V, I, O, T) \in State\_set \times State \)
\(\times Variable\_set \times Input\_set \times Output\_set \times \)
\( Transition\_set \).
-
S
:
-
is a finite set of \( State \)s,
-
\(s_0 \in S\)
:
-
is an initial \( State \),
-
V
:
-
is a finite set of \( Variable \)s,
-
I
:
-
is a finite set of \( Input \)s,
-
O
:
-
is a finite set of \( Output \)s,
-
T
:
-
is a finite set of transitions, \(t \in T\) can be defined as
a 5-tuple \((s, i, g, op, s')\),
-
s
:
-
is the source \( State \),
-
i
:
-
is an \( Input \),
-
g
:
-
is a guard of the form \(e \circ e'\), where e and \(e'\) are
algebraic expressions and \(\circ \in \{<,>,\ne ,=,\le ,\ge \}\),
-
op
:
-
is a sequence of output and assignment operations of the form \(v=e\) with
-
\(v\in V\)
:
-
and e is an expression,
-
\(s'\)
:
-
is the target State [17].
An example EFSM is presented in Sect. 4.2 in Fig. 4. Semantically, a guard g is a Boolean function that takes the variable valuations v as input and returns a Boolean value. An operation op is a function mapping the current variable valuations to a pair of new valuations and an optional output \(o \in O\).
In order to perform PBT for an EFSM, a state machine specification \( spec \) has to be provided. This \( spec \)ification includes functions to set the initial state of the model and the SUT, a set of commands \( cmds \) and a \( next \) function that builds a command generator \( CmdGen \) for a given model state:
Definition 4
(State machine specification)
Algorithm 3 in Sect. 4.2 outlines an example specification for the incident manager as it is required for FsCheck.
A \( Model \) object consists of fields representing the current EFSM state s, the valuations for the variables v, the transition set T, the last output o and a \( doStep \) function that performs the execution of a transition.
Note that the SUT is defined in the same way as the model and is, therefore, omitted.
A command \(Cmd_{in} \in spec.cmds\) encodes a set of transitions \(T_{in}\) with the same input in. They encapsulate preconditions, postconditions and the execution semantics of these transitions. Preconditions \( pre \) define the permitted transition sequences by enabling the command only in states where the input in is allowed. Postcondition \( post \) can verify the effects of the command, e.g. by comparing the state of the model and the SUT. The execution semantics are encoded via the functions \( runModel \) and \( runActual \) for executing the Model and the SUT. The definition of a command is shown in Fig. 3. Note that in this definition we show various possible checks in the postcondition, i.e. we analyse the current state of the SUT, variable valuations and the output. In reality this may not be feasible, because the SUT might not provide all this information. Hence, in many cases it may only be possible to check the output in the postcondition. An example implementation of a command is presented in Sect. 4.2 in Algorithm 4. This example demonstrates the function definitions for an IncidentCreateTask.
A property of an EFSM is that for each permitted path, the postcondition of each transition, respectively, command of the path must hold. In order to verify this property, a PBT tool produces random command sequences and checks the postconditions after each command execution.
The state machine property must hold in all settings of the model \( Model\_set \) and the SUT \( SUT\_set \) that are reachable via valid command sequences. A command sequence is valid if all its preconditions are satisfied. Hence, given a specification \( spec \), a state machine property for EFSMs can be defined as follows:
Definition 5
(State machine property)
$$\begin{aligned} \begin{aligned}&\forall cmd_{in} \in spec.cmds, model \in Model\_set : \\&\exists sut \in SUT\_set : cmd_{in}.pre(model) \implies \\&\qquad \qquad cmd_{in}.post( cmd_{in}.runModel (model), \\&\qquad \qquad \qquad \qquad \qquad cmd_{in}.post( cmd_{in}.runActual (sut))\\ \end{aligned} \end{aligned}$$
Algorithm 1 shows the pseudocode of the test case generation for such a property. The algorithm takes a \( spec \) and a \( size \) parameter for the length of the test case as input and returns a \( testSequence \), which is a sequence of \( (Cmd, Model) \) pairs. In the first step, the initial model is created with the \( initialModel \) function of the \( spec \). Next, there is a loop over the size parameter. In each iteration, a command generator gen is built with the \( next \) function of the \( spec \). This function takes the model (Line 9) and creates a subset of all commands by checking their precondition. The function returns an \( Elements \) generator, which selects one element of this set with a uniform distribution (Line 11). The sample function of this generator is called to produce a command (Line 4). This command is executed with \( runModel \), which returns a new model that incorporates the state change. Note that we need a new model and not only change the current one, because future changes should not affect the old stored model instances. This new model and the command are stored in the \( testSequence \). Finally, after the loop is finished we return the \( testSequence \), which represents a test case.
Algorithm 2 shows how such a generated test case can be executed. The test case is the input of this algorithm together with a \( spec \), and the result is a verdict. (Note that shrinking is omitted in this simplified algorithm.) In the first step, the initial SUT is built by the \( initialActual \) function of the \( spec \). After that we loop over the \( testSequence \). Next, the command is executed on the SUT with \( runActual \), which results in a modified SUT (Line 3). The postcondition of the command is applied to compare the SUT with the stored model of the \( testSequence \). If it is false, then the test failed. Otherwise, the execution continues and if the loop is finished, then the postconditions of all commands were satisfied and a pass-verdict is returned.
Example of model-based testing with FsCheck
In this subsection, we show how FsCheck can be applied for model-based testing. A simple example of an incident manager taken from our industrial case study shall serve to demonstrate how the necessary interface implementations have to be realised.
FsCheck modelling In order to use FsCheck for model-based testing, we need a specification class that implements an ICommandGenerator interface and contains the following elements:
-
SUT definition (which is called Actual by FsCheck)
-
Model definition
-
Initial state of the SUT and the model
-
Generator for the next command given the current state of the model
-
Commands combining preconditions, postconditions and the transition execution semantics of the SUT and the model
Details about the structure of such specifications were already presented in Sect. 4.1. Now we give a more concrete example for FsCheck on an object-oriented level. Algorithm 3 outlines an example specification. In order to implement the interface for FsCheck, we need the mentioned elements. The class of the SUT is basically a wrapper that provides methods for the execution of all tasks and a method to retrieve the current state of one incident object of the SUT. An incident object is an element of the application domain. For example, it could be a bug report. It has a number of attributes (form data), which are stored in the database. In this example, we assume that the attributes are set statically in the wrapper class of the SUT. In Sect. 6, we will see, how form data can be generated automatically for these attributes.
Figure 4 illustrates the state machine of one incident object. Initially, the machine is in a global state. The IncidentCreateTask (abbreviated as Create) creates and opens a new incident object, which can be edited and closed with the corresponding tasks. The transitions are labelled as follows: input i, an optional guard g/assignment operations op, and an output o. The assignment operations of this EFSM assign values to the attribute variables, and the output indicates the target state of a transition. The initial global state has a special meaning: tasks of the global state, i.e. IncidentCreateTask, are globally enabled in all states. Hence, it is possible in every state to create new incident objects. However, to simplify the discussion, we assume that the state machine only represents a currently opened incident object. Generally, in an object-oriented system comprising several objects, we need functionality to switch between active objects. This functionality is discussed in Sect. 5.2.
The initial states of the model and SUT are set by creating new objects (Algorithm 3: Lines 2 and 5). The generator in the \( next \) function selects one element of a command set randomly, which can be accomplished with the default Elements generator of FsCheck (Line 9).
In the standard PBT approach, all command classes need to be defined manually as shown in Algorithm 4. The classes need to define how the transitions should be executed on the model and SUT and what postcondition should hold after the execution. In this simple example, the execution of the model only changes the state, later we will also see how we handle form data. For example, the state-changing function of an IncidentCreateTask is defined as follows:
Note that in contrast to the previous abstract definition, the postcondition here checks whether the state of the SUT matches the state of the model. Moreover, a \( toString \) method can be used to display various information of the command and optionally a precondition can be defined. The classes for the IncidentEditTask and the IncidentCloseTask command are similar to this class and are, therefore, omitted.
For a large model with many transitions, it is not practical that all commands have a separate class. Therefore, it makes sense to implement this definition in a more generic way for all possible transitions and to automate the process as far as possible.
Command generation and execution The tool FsCheck generates test cases according to Algorithm 1 with the difference that the specification is provided in an object-oriented style as shown in Algorithm 3. After a test case is generated, it is executed on the SUT and the state of the SUT is compared with the stored model state after each command execution as explained in Algorithm 2.
In order to start testing in FsCheck, the specification has to be converted into a property. This is achieved with the \( toProperty() \) method of FsCheck. The property can then be tested by calling the QuickCheck() method or also with the help of unit testing frameworks:
By default, 100 test cases will be generated and executed, but this number can be configured. Listing 2 shows two example sequences that were produced by FsCheck for the incident specification. It can be seen that the sequences have quite different lengths, because FsCheck generates them randomly with a variety of lengths. Moreover, FsCheck classifies the sequences according to their lengths, which can be seen in the last line of the listing. These classifications can be helpful to find out that a certain generator only considers trivial cases. Each of these generated tasks in the command sequences requires form data for the attributes, which also needs to be generated. Listing 3 shows example form data for some attributes that was generated randomly for the IncidentCreateTask. Note that this randomly generated strings form a kind of robustness test in order to check that the SUT can process non-standard input.